[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250831 07:57:06.714879 30564 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.217.62:34243
I20250831 07:57:06.716423 30564 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 07:57:06.717495 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 07:57:06.734215 30571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:57:06.734236 30573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:57:06.735740 30564 server_base.cc:1047] running on GCE node
W20250831 07:57:06.734242 30570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:57:07.928516 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 07:57:07.928786 30564 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 07:57:07.928982 30564 hybrid_clock.cc:648] HybridClock initialized: now 1756627027928949 us; error 0 us; skew 500 ppm
I20250831 07:57:07.929826 30564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 07:57:07.936118 30564 webserver.cc:480] Webserver started at http://127.29.217.62:36553/ using document root <none> and password file <none>
I20250831 07:57:07.936995 30564 fs_manager.cc:362] Metadata directory not provided
I20250831 07:57:07.937252 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 07:57:07.937681 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 07:57:07.943697 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/master-0-root/instance:
uuid: "cdc80dd77c5d4b50a69ab18e290f99a1"
format_stamp: "Formatted at 2025-08-31 07:57:07 on dist-test-slave-87tg"
I20250831 07:57:07.951610 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250831 07:57:07.956838 30579 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:07.958024 30564 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250831 07:57:07.958390 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/master-0-root
uuid: "cdc80dd77c5d4b50a69ab18e290f99a1"
format_stamp: "Formatted at 2025-08-31 07:57:07 on dist-test-slave-87tg"
I20250831 07:57:07.958753 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 07:57:08.018278 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 07:57:08.019783 30564 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 07:57:08.020246 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 07:57:08.090900 30564 rpc_server.cc:307] RPC server started. Bound to: 127.29.217.62:34243
I20250831 07:57:08.090976 30630 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.217.62:34243 every 8 connection(s)
I20250831 07:57:08.098649 30631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:08.120534 30631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1: Bootstrap starting.
I20250831 07:57:08.126349 30631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:08.128230 30631 log.cc:826] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1: Log is configured to *not* fsync() on all Append() calls
I20250831 07:57:08.133216 30631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1: No bootstrap required, opened a new log
I20250831 07:57:08.153641 30631 raft_consensus.cc:357] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc80dd77c5d4b50a69ab18e290f99a1" member_type: VOTER }
I20250831 07:57:08.154181 30631 raft_consensus.cc:383] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:08.154448 30631 raft_consensus.cc:738] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc80dd77c5d4b50a69ab18e290f99a1, State: Initialized, Role: FOLLOWER
I20250831 07:57:08.155318 30631 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc80dd77c5d4b50a69ab18e290f99a1" member_type: VOTER }
I20250831 07:57:08.155844 30631 raft_consensus.cc:397] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250831 07:57:08.156132 30631 raft_consensus.cc:491] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250831 07:57:08.156464 30631 raft_consensus.cc:3058] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:08.160537 30631 raft_consensus.cc:513] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc80dd77c5d4b50a69ab18e290f99a1" member_type: VOTER }
I20250831 07:57:08.161223 30631 leader_election.cc:304] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc80dd77c5d4b50a69ab18e290f99a1; no voters:
I20250831 07:57:08.162897 30631 leader_election.cc:290] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250831 07:57:08.163257 30634 raft_consensus.cc:2802] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:08.165859 30634 raft_consensus.cc:695] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [term 1 LEADER]: Becoming Leader. State: Replica: cdc80dd77c5d4b50a69ab18e290f99a1, State: Running, Role: LEADER
I20250831 07:57:08.167224 30634 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc80dd77c5d4b50a69ab18e290f99a1" member_type: VOTER }
I20250831 07:57:08.167975 30631 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 07:57:08.179378 30635 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdc80dd77c5d4b50a69ab18e290f99a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc80dd77c5d4b50a69ab18e290f99a1" member_type: VOTER } }
I20250831 07:57:08.179948 30635 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [sys.catalog]: This master's current role is: LEADER
I20250831 07:57:08.182293 30636 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc80dd77c5d4b50a69ab18e290f99a1. Latest consensus state: current_term: 1 leader_uuid: "cdc80dd77c5d4b50a69ab18e290f99a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc80dd77c5d4b50a69ab18e290f99a1" member_type: VOTER } }
I20250831 07:57:08.182920 30636 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [sys.catalog]: This master's current role is: LEADER
I20250831 07:57:08.183048 30642 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250831 07:57:08.195745 30642 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250831 07:57:08.202751 30564 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250831 07:57:08.217269 30642 catalog_manager.cc:1349] Generated new cluster ID: 11ac8678f49d4fe4a71dadbfe33e1d06
I20250831 07:57:08.217562 30642 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250831 07:57:08.237561 30642 catalog_manager.cc:1372] Generated new certificate authority record
I20250831 07:57:08.239110 30642 catalog_manager.cc:1506] Loading token signing keys...
I20250831 07:57:08.259735 30642 catalog_manager.cc:5955] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1: Generated new TSK 0
I20250831 07:57:08.260735 30642 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250831 07:57:08.273978 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 07:57:08.284459 30653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:57:08.286554 30654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:57:08.290472 30564 server_base.cc:1047] running on GCE node
W20250831 07:57:08.290796 30656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:57:08.291675 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 07:57:08.291913 30564 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 07:57:08.292083 30564 hybrid_clock.cc:648] HybridClock initialized: now 1756627028292062 us; error 0 us; skew 500 ppm
I20250831 07:57:08.292670 30564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 07:57:08.296787 30564 webserver.cc:480] Webserver started at http://127.29.217.1:40869/ using document root <none> and password file <none>
I20250831 07:57:08.297343 30564 fs_manager.cc:362] Metadata directory not provided
I20250831 07:57:08.297530 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 07:57:08.297821 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 07:57:08.298938 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-0-root/instance:
uuid: "357750ee400f42f8a09a88487e8bd247"
format_stamp: "Formatted at 2025-08-31 07:57:08 on dist-test-slave-87tg"
I20250831 07:57:08.305337 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250831 07:57:08.310427 30661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:08.311360 30564 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250831 07:57:08.311735 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-0-root
uuid: "357750ee400f42f8a09a88487e8bd247"
format_stamp: "Formatted at 2025-08-31 07:57:08 on dist-test-slave-87tg"
I20250831 07:57:08.312165 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 07:57:08.329133 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 07:57:08.330948 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 07:57:08.334270 30564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 07:57:08.340310 30564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 07:57:08.340668 30564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:08.340997 30564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 07:57:08.341223 30564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:08.453791 30723 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.217.1:33467 every 8 connection(s)
I20250831 07:57:08.453820 30564 rpc_server.cc:307] RPC server started. Bound to: 127.29.217.1:33467
I20250831 07:57:08.462642 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 07:57:08.479768 30728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:57:08.482988 30729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:57:08.494360 30731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:57:08.494731 30564 server_base.cc:1047] running on GCE node
I20250831 07:57:08.496094 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 07:57:08.496384 30564 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 07:57:08.496598 30564 hybrid_clock.cc:648] HybridClock initialized: now 1756627028496583 us; error 0 us; skew 500 ppm
I20250831 07:57:08.497568 30564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 07:57:08.500398 30724 heartbeater.cc:344] Connected to a master server at 127.29.217.62:34243
I20250831 07:57:08.500834 30724 heartbeater.cc:461] Registering TS with master...
I20250831 07:57:08.501706 30564 webserver.cc:480] Webserver started at http://127.29.217.2:43711/ using document root <none> and password file <none>
I20250831 07:57:08.501835 30724 heartbeater.cc:507] Master 127.29.217.62:34243 requested a full tablet report, sending...
I20250831 07:57:08.502398 30564 fs_manager.cc:362] Metadata directory not provided
I20250831 07:57:08.502660 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 07:57:08.503011 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 07:57:08.504483 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-1-root/instance:
uuid: "39d7bd8c4f38482b8548151b5f717bcf"
format_stamp: "Formatted at 2025-08-31 07:57:08 on dist-test-slave-87tg"
I20250831 07:57:08.505334 30596 ts_manager.cc:194] Registered new tserver with Master: 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467)
I20250831 07:57:08.507417 30596 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51910
I20250831 07:57:08.511759 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20250831 07:57:08.517187 30736 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:08.518029 30564 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250831 07:57:08.518355 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-1-root
uuid: "39d7bd8c4f38482b8548151b5f717bcf"
format_stamp: "Formatted at 2025-08-31 07:57:08 on dist-test-slave-87tg"
I20250831 07:57:08.518723 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 07:57:08.561971 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 07:57:08.563768 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 07:57:08.565876 30564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 07:57:08.569200 30564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 07:57:08.569409 30564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:08.569684 30564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 07:57:08.569885 30564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:08.680970 30564 rpc_server.cc:307] RPC server started. Bound to: 127.29.217.2:41785
I20250831 07:57:08.681035 30798 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.217.2:41785 every 8 connection(s)
I20250831 07:57:08.685909 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 07:57:08.698988 30802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:57:08.699877 30803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:57:08.702800 30799 heartbeater.cc:344] Connected to a master server at 127.29.217.62:34243
I20250831 07:57:08.703312 30799 heartbeater.cc:461] Registering TS with master...
I20250831 07:57:08.704438 30799 heartbeater.cc:507] Master 127.29.217.62:34243 requested a full tablet report, sending...
I20250831 07:57:08.707248 30596 ts_manager.cc:194] Registered new tserver with Master: 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:08.707468 30564 server_base.cc:1047] running on GCE node
W20250831 07:57:08.708750 30805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:57:08.708916 30596 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51918
I20250831 07:57:08.709839 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 07:57:08.710091 30564 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 07:57:08.710275 30564 hybrid_clock.cc:648] HybridClock initialized: now 1756627028710261 us; error 0 us; skew 500 ppm
I20250831 07:57:08.711014 30564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 07:57:08.713819 30564 webserver.cc:480] Webserver started at http://127.29.217.3:36181/ using document root <none> and password file <none>
I20250831 07:57:08.714430 30564 fs_manager.cc:362] Metadata directory not provided
I20250831 07:57:08.714686 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 07:57:08.714990 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 07:57:08.716380 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-2-root/instance:
uuid: "4598d92c93d34ad5b95e906a309af48e"
format_stamp: "Formatted at 2025-08-31 07:57:08 on dist-test-slave-87tg"
I20250831 07:57:08.722426 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250831 07:57:08.726796 30810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:08.727684 30564 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250831 07:57:08.728014 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-2-root
uuid: "4598d92c93d34ad5b95e906a309af48e"
format_stamp: "Formatted at 2025-08-31 07:57:08 on dist-test-slave-87tg"
I20250831 07:57:08.728278 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756627026548763-30564-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 07:57:08.742700 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 07:57:08.743947 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 07:57:08.746016 30564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 07:57:08.749207 30564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 07:57:08.749415 30564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:08.749655 30564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 07:57:08.749816 30564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:08.862236 30564 rpc_server.cc:307] RPC server started. Bound to: 127.29.217.3:43897
I20250831 07:57:08.862371 30872 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.217.3:43897 every 8 connection(s)
I20250831 07:57:08.879601 30873 heartbeater.cc:344] Connected to a master server at 127.29.217.62:34243
I20250831 07:57:08.880076 30873 heartbeater.cc:461] Registering TS with master...
I20250831 07:57:08.881011 30873 heartbeater.cc:507] Master 127.29.217.62:34243 requested a full tablet report, sending...
I20250831 07:57:08.883019 30596 ts_manager.cc:194] Registered new tserver with Master: 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:08.883448 30564 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013676296s
I20250831 07:57:08.884680 30596 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51930
I20250831 07:57:08.888238 30564 test_util.cc:276] Using random seed: -1710273904
I20250831 07:57:08.945019 30596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51938:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250831 07:57:08.949026 30596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250831 07:57:09.068890 30681 tablet_service.cc:1468] Processing CreateTablet for tablet 68688b732e6a47858fa4d8257adf011f (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250831 07:57:09.068972 30688 tablet_service.cc:1468] Processing CreateTablet for tablet 47a48a1d49224a53a0949ef880502472 (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250831 07:57:09.068099 30687 tablet_service.cc:1468] Processing CreateTablet for tablet fffdf93425de4d52b8d33fd7cb7ecb9a (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250831 07:57:09.070017 30684 tablet_service.cc:1468] Processing CreateTablet for tablet 27b19174fafc4d6bb0f98dc93d13dda4 (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250831 07:57:09.069790 30685 tablet_service.cc:1468] Processing CreateTablet for tablet 8524e0bbc5a04120a1862c3f2b492c3f (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250831 07:57:09.070147 30686 tablet_service.cc:1468] Processing CreateTablet for tablet 24dbfaab9a7a42ffa64a24fa6f384740 (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250831 07:57:09.075268 30681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68688b732e6a47858fa4d8257adf011f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.065475 30689 tablet_service.cc:1468] Processing CreateTablet for tablet 1c786bfc839247baafd87289b48ca14b (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250831 07:57:09.073168 30682 tablet_service.cc:1468] Processing CreateTablet for tablet 7fbb315d02b6426b8531915f088cce66 (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250831 07:57:09.066567 30683 tablet_service.cc:1468] Processing CreateTablet for tablet d0b161ad222348bea9cf99d0d2dc923d (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250831 07:57:09.086504 30689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c786bfc839247baafd87289b48ca14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.087611 30687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fffdf93425de4d52b8d33fd7cb7ecb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.089299 30683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b161ad222348bea9cf99d0d2dc923d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.092134 30688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a48a1d49224a53a0949ef880502472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.096525 30682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fbb315d02b6426b8531915f088cce66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.097604 30684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b19174fafc4d6bb0f98dc93d13dda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.106812 30686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24dbfaab9a7a42ffa64a24fa6f384740. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.110265 30764 tablet_service.cc:1468] Processing CreateTablet for tablet 1c786bfc839247baafd87289b48ca14b (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250831 07:57:09.110872 30763 tablet_service.cc:1468] Processing CreateTablet for tablet 47a48a1d49224a53a0949ef880502472 (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250831 07:57:09.120388 30763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a48a1d49224a53a0949ef880502472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.120446 30685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8524e0bbc5a04120a1862c3f2b492c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.110824 30762 tablet_service.cc:1468] Processing CreateTablet for tablet fffdf93425de4d52b8d33fd7cb7ecb9a (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250831 07:57:09.113890 30761 tablet_service.cc:1468] Processing CreateTablet for tablet 24dbfaab9a7a42ffa64a24fa6f384740 (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250831 07:57:09.129761 30764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c786bfc839247baafd87289b48ca14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.127964 30759 tablet_service.cc:1468] Processing CreateTablet for tablet 27b19174fafc4d6bb0f98dc93d13dda4 (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250831 07:57:09.125267 30760 tablet_service.cc:1468] Processing CreateTablet for tablet 8524e0bbc5a04120a1862c3f2b492c3f (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250831 07:57:09.138521 30758 tablet_service.cc:1468] Processing CreateTablet for tablet d0b161ad222348bea9cf99d0d2dc923d (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250831 07:57:09.139206 30762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fffdf93425de4d52b8d33fd7cb7ecb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.137657 30756 tablet_service.cc:1468] Processing CreateTablet for tablet 68688b732e6a47858fa4d8257adf011f (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250831 07:57:09.145077 30757 tablet_service.cc:1468] Processing CreateTablet for tablet 7fbb315d02b6426b8531915f088cce66 (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250831 07:57:09.160733 30758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b161ad222348bea9cf99d0d2dc923d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.162410 30759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b19174fafc4d6bb0f98dc93d13dda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.176087 30757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fbb315d02b6426b8531915f088cce66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.184322 30760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8524e0bbc5a04120a1862c3f2b492c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.192997 30756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68688b732e6a47858fa4d8257adf011f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.201360 30838 tablet_service.cc:1468] Processing CreateTablet for tablet 1c786bfc839247baafd87289b48ca14b (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250831 07:57:09.202589 30837 tablet_service.cc:1468] Processing CreateTablet for tablet 47a48a1d49224a53a0949ef880502472 (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250831 07:57:09.204929 30761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24dbfaab9a7a42ffa64a24fa6f384740. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.212139 30838 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c786bfc839247baafd87289b48ca14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.228921 30834 tablet_service.cc:1468] Processing CreateTablet for tablet 8524e0bbc5a04120a1862c3f2b492c3f (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250831 07:57:09.211886 30836 tablet_service.cc:1468] Processing CreateTablet for tablet fffdf93425de4d52b8d33fd7cb7ecb9a (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250831 07:57:09.245712 30835 tablet_service.cc:1468] Processing CreateTablet for tablet 24dbfaab9a7a42ffa64a24fa6f384740 (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250831 07:57:09.248636 30833 tablet_service.cc:1468] Processing CreateTablet for tablet 27b19174fafc4d6bb0f98dc93d13dda4 (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250831 07:57:09.254281 30837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a48a1d49224a53a0949ef880502472. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 07:57:09.273432 30644 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 07:57:09.273998 30891 tablet_bootstrap.cc:492] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247: Bootstrap starting.
I20250831 07:57:09.273590 30832 tablet_service.cc:1468] Processing CreateTablet for tablet d0b161ad222348bea9cf99d0d2dc923d (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250831 07:57:09.291384 30834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8524e0bbc5a04120a1862c3f2b492c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.292713 30832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b161ad222348bea9cf99d0d2dc923d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.285933 30830 tablet_service.cc:1468] Processing CreateTablet for tablet 68688b732e6a47858fa4d8257adf011f (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250831 07:57:09.297508 30831 tablet_service.cc:1468] Processing CreateTablet for tablet 7fbb315d02b6426b8531915f088cce66 (DEFAULT_TABLE table=test-workload [id=e12ef276dfdf4e4983e0a1a5eced76d1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250831 07:57:09.318400 30833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b19174fafc4d6bb0f98dc93d13dda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.326941 30836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fffdf93425de4d52b8d33fd7cb7ecb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.330384 30831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fbb315d02b6426b8531915f088cce66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.336833 30830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68688b732e6a47858fa4d8257adf011f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.385217 30835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24dbfaab9a7a42ffa64a24fa6f384740. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:09.388427 30892 tablet_bootstrap.cc:492] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf: Bootstrap starting.
I20250831 07:57:09.406400 30892 tablet_bootstrap.cc:654] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.412235 30893 tablet_bootstrap.cc:492] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e: Bootstrap starting.
I20250831 07:57:09.415184 30891 tablet_bootstrap.cc:654] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.453557 30892 tablet_bootstrap.cc:492] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf: No bootstrap required, opened a new log
I20250831 07:57:09.454154 30892 ts_tablet_manager.cc:1397] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf: Time spent bootstrapping tablet: real 0.080s user 0.027s sys 0.017s
I20250831 07:57:09.469995 30891 tablet_bootstrap.cc:492] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247: No bootstrap required, opened a new log
I20250831 07:57:09.470471 30891 ts_tablet_manager.cc:1397] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247: Time spent bootstrapping tablet: real 0.237s user 0.016s sys 0.018s
I20250831 07:57:09.470237 30892 raft_consensus.cc:357] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.470886 30892 raft_consensus.cc:383] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.471693 30892 raft_consensus.cc:738] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.472457 30892 consensus_queue.cc:260] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.479854 30891 raft_consensus.cc:357] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.480962 30891 raft_consensus.cc:383] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.481331 30891 raft_consensus.cc:738] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.486009 30893 tablet_bootstrap.cc:654] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.485821 30891 consensus_queue.cc:260] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.494115 30892 ts_tablet_manager.cc:1428] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf: Time spent starting tablet: real 0.035s user 0.014s sys 0.009s
I20250831 07:57:09.495162 30799 heartbeater.cc:499] Master 127.29.217.62:34243 was elected leader, sending a full tablet report...
I20250831 07:57:09.499648 30892 tablet_bootstrap.cc:492] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf: Bootstrap starting.
I20250831 07:57:09.500991 30891 ts_tablet_manager.cc:1428] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247: Time spent starting tablet: real 0.030s user 0.012s sys 0.004s
I20250831 07:57:09.510952 30892 tablet_bootstrap.cc:654] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.514402 30724 heartbeater.cc:499] Master 127.29.217.62:34243 was elected leader, sending a full tablet report...
I20250831 07:57:09.516616 30891 tablet_bootstrap.cc:492] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247: Bootstrap starting.
I20250831 07:57:09.523954 30891 tablet_bootstrap.cc:654] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.530520 30893 tablet_bootstrap.cc:492] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e: No bootstrap required, opened a new log
I20250831 07:57:09.531026 30893 ts_tablet_manager.cc:1397] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e: Time spent bootstrapping tablet: real 0.130s user 0.020s sys 0.012s
I20250831 07:57:09.531071 30892 tablet_bootstrap.cc:492] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf: No bootstrap required, opened a new log
I20250831 07:57:09.531677 30892 ts_tablet_manager.cc:1397] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.002s
I20250831 07:57:09.534072 30893 raft_consensus.cc:357] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.534806 30893 raft_consensus.cc:383] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.535102 30893 raft_consensus.cc:738] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.535216 30892 raft_consensus.cc:357] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.536082 30892 raft_consensus.cc:383] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.535837 30893 consensus_queue.cc:260] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.536381 30892 raft_consensus.cc:738] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.537092 30892 consensus_queue.cc:260] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.541406 30892 ts_tablet_manager.cc:1428] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20250831 07:57:09.543493 30892 tablet_bootstrap.cc:492] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf: Bootstrap starting.
I20250831 07:57:09.551340 30892 tablet_bootstrap.cc:654] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.553371 30893 ts_tablet_manager.cc:1428] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20250831 07:57:09.554499 30893 tablet_bootstrap.cc:492] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e: Bootstrap starting.
I20250831 07:57:09.555397 30891 tablet_bootstrap.cc:492] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247: No bootstrap required, opened a new log
I20250831 07:57:09.555867 30891 ts_tablet_manager.cc:1397] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.010s
I20250831 07:57:09.558565 30891 raft_consensus.cc:357] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.559290 30891 raft_consensus.cc:383] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.559576 30891 raft_consensus.cc:738] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.560181 30873 heartbeater.cc:499] Master 127.29.217.62:34243 was elected leader, sending a full tablet report...
I20250831 07:57:09.562047 30892 tablet_bootstrap.cc:492] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf: No bootstrap required, opened a new log
I20250831 07:57:09.562358 30892 ts_tablet_manager.cc:1397] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.002s
I20250831 07:57:09.565205 30893 tablet_bootstrap.cc:654] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.565161 30892 raft_consensus.cc:357] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.565984 30892 raft_consensus.cc:383] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.566298 30892 raft_consensus.cc:738] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.567024 30892 consensus_queue.cc:260] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.572170 30891 consensus_queue.cc:260] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.573009 30892 ts_tablet_manager.cc:1428] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250831 07:57:09.575089 30891 ts_tablet_manager.cc:1428] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20250831 07:57:09.576108 30891 tablet_bootstrap.cc:492] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247: Bootstrap starting.
I20250831 07:57:09.576859 30892 tablet_bootstrap.cc:492] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf: Bootstrap starting.
I20250831 07:57:09.577421 30893 tablet_bootstrap.cc:492] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e: No bootstrap required, opened a new log
I20250831 07:57:09.577930 30893 ts_tablet_manager.cc:1397] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.000s
I20250831 07:57:09.581279 30893 raft_consensus.cc:357] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.582238 30893 raft_consensus.cc:383] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.582583 30893 raft_consensus.cc:738] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.583340 30893 consensus_queue.cc:260] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.584021 30892 tablet_bootstrap.cc:654] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.586232 30893 ts_tablet_manager.cc:1428] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20250831 07:57:09.587236 30893 tablet_bootstrap.cc:492] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e: Bootstrap starting.
I20250831 07:57:09.593955 30891 tablet_bootstrap.cc:654] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.594478 30893 tablet_bootstrap.cc:654] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.605969 30892 tablet_bootstrap.cc:492] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf: No bootstrap required, opened a new log
I20250831 07:57:09.606487 30892 ts_tablet_manager.cc:1397] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.016s
I20250831 07:57:09.609609 30892 raft_consensus.cc:357] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.610301 30892 raft_consensus.cc:383] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.610592 30892 raft_consensus.cc:738] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.611239 30892 consensus_queue.cc:260] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.613615 30893 tablet_bootstrap.cc:492] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e: No bootstrap required, opened a new log
I20250831 07:57:09.613889 30892 ts_tablet_manager.cc:1428] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250831 07:57:09.614034 30893 ts_tablet_manager.cc:1397] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.000s
I20250831 07:57:09.614912 30892 tablet_bootstrap.cc:492] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf: Bootstrap starting.
I20250831 07:57:09.617353 30893 raft_consensus.cc:357] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.618096 30893 raft_consensus.cc:383] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.618360 30893 raft_consensus.cc:738] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.619047 30893 consensus_queue.cc:260] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
W20250831 07:57:09.621068 30874 tablet.cc:2378] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250831 07:57:09.621526 30893 ts_tablet_manager.cc:1428] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20250831 07:57:09.621834 30892 tablet_bootstrap.cc:654] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.622402 30893 tablet_bootstrap.cc:492] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e: Bootstrap starting.
I20250831 07:57:09.630086 30893 tablet_bootstrap.cc:654] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.630466 30897 raft_consensus.cc:491] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:09.631203 30897 raft_consensus.cc:513] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.632408 30896 raft_consensus.cc:491] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:09.632902 30896 raft_consensus.cc:513] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.635408 30896 leader_election.cc:290] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:09.639539 30897 leader_election.cc:290] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:09.642956 30891 tablet_bootstrap.cc:492] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247: No bootstrap required, opened a new log
I20250831 07:57:09.643481 30891 ts_tablet_manager.cc:1397] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.018s
I20250831 07:57:09.651741 30891 raft_consensus.cc:357] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.653033 30891 raft_consensus.cc:383] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.653462 30891 raft_consensus.cc:738] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.664327 30891 consensus_queue.cc:260] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.670324 30891 ts_tablet_manager.cc:1428] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247: Time spent starting tablet: real 0.026s user 0.008s sys 0.001s
I20250831 07:57:09.675238 30891 tablet_bootstrap.cc:492] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247: Bootstrap starting.
I20250831 07:57:09.698451 30892 tablet_bootstrap.cc:492] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf: No bootstrap required, opened a new log
I20250831 07:57:09.698980 30892 ts_tablet_manager.cc:1397] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf: Time spent bootstrapping tablet: real 0.084s user 0.031s sys 0.032s
I20250831 07:57:09.703459 30892 raft_consensus.cc:357] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.704236 30892 raft_consensus.cc:383] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.704535 30892 raft_consensus.cc:738] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.705221 30892 consensus_queue.cc:260] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.709331 30891 tablet_bootstrap.cc:654] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.713505 30893 tablet_bootstrap.cc:492] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e: No bootstrap required, opened a new log
I20250831 07:57:09.715210 30893 ts_tablet_manager.cc:1397] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e: Time spent bootstrapping tablet: real 0.093s user 0.015s sys 0.019s
I20250831 07:57:09.719404 30892 ts_tablet_manager.cc:1428] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf: Time spent starting tablet: real 0.020s user 0.007s sys 0.002s
I20250831 07:57:09.720683 30848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4598d92c93d34ad5b95e906a309af48e" is_pre_election: true
W20250831 07:57:09.724296 30665 leader_election.cc:343] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:09.726172 30892 tablet_bootstrap.cc:492] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf: Bootstrap starting.
I20250831 07:57:09.731324 30893 raft_consensus.cc:357] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.732790 30893 raft_consensus.cc:383] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.733163 30893 raft_consensus.cc:738] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.737373 30848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b161ad222348bea9cf99d0d2dc923d" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4598d92c93d34ad5b95e906a309af48e" is_pre_election: true
I20250831 07:57:09.738016 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b161ad222348bea9cf99d0d2dc923d" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247" is_pre_election: true
I20250831 07:57:09.739110 30699 raft_consensus.cc:2466] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 0.
I20250831 07:57:09.735724 30893 consensus_queue.cc:260] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
W20250831 07:57:09.740480 30740 leader_election.cc:343] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:09.741165 30891 tablet_bootstrap.cc:492] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247: No bootstrap required, opened a new log
I20250831 07:57:09.741593 30892 tablet_bootstrap.cc:654] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.741640 30891 ts_tablet_manager.cc:1397] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247: Time spent bootstrapping tablet: real 0.067s user 0.018s sys 0.002s
I20250831 07:57:09.743851 30738 leader_election.cc:304] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 39d7bd8c4f38482b8548151b5f717bcf; no voters: 4598d92c93d34ad5b95e906a309af48e
I20250831 07:57:09.744684 30896 raft_consensus.cc:2802] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:09.745043 30896 raft_consensus.cc:491] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:09.745260 30893 ts_tablet_manager.cc:1428] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e: Time spent starting tablet: real 0.030s user 0.010s sys 0.000s
I20250831 07:57:09.745484 30896 raft_consensus.cc:3058] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:09.746454 30893 tablet_bootstrap.cc:492] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e: Bootstrap starting.
I20250831 07:57:09.749281 30901 raft_consensus.cc:491] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:09.751012 30901 raft_consensus.cc:513] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.752144 30891 raft_consensus.cc:357] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.754956 30891 raft_consensus.cc:383] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.755249 30891 raft_consensus.cc:738] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.756099 30891 consensus_queue.cc:260] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.769537 30901 leader_election.cc:290] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:09.772519 30896 raft_consensus.cc:513] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.776213 30896 leader_election.cc:290] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:09.776752 30896 raft_consensus.cc:491] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:09.777168 30896 raft_consensus.cc:513] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.778962 30896 leader_election.cc:290] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:09.783313 30698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8524e0bbc5a04120a1862c3f2b492c3f" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247" is_pre_election: true
I20250831 07:57:09.784040 30848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b161ad222348bea9cf99d0d2dc923d" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:09.784802 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b161ad222348bea9cf99d0d2dc923d" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:09.785357 30699 raft_consensus.cc:3058] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:09.785715 30848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8524e0bbc5a04120a1862c3f2b492c3f" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4598d92c93d34ad5b95e906a309af48e" is_pre_election: true
W20250831 07:57:09.787124 30740 leader_election.cc:343] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:09.788251 30848 raft_consensus.cc:2466] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 0.
I20250831 07:57:09.789739 30891 ts_tablet_manager.cc:1428] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247: Time spent starting tablet: real 0.048s user 0.006s sys 0.001s
I20250831 07:57:09.792030 30740 leader_election.cc:304] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d7bd8c4f38482b8548151b5f717bcf, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:09.793062 30891 tablet_bootstrap.cc:492] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247: Bootstrap starting.
I20250831 07:57:09.795722 30917 raft_consensus.cc:2802] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:09.796165 30917 raft_consensus.cc:491] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:09.796509 30917 raft_consensus.cc:3058] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:57:09.801837 30738 leader_election.cc:343] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:09.801393 30774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf" is_pre_election: true
I20250831 07:57:09.804589 30892 tablet_bootstrap.cc:492] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf: No bootstrap required, opened a new log
I20250831 07:57:09.806454 30892 ts_tablet_manager.cc:1397] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.020s
W20250831 07:57:09.808401 30662 leader_election.cc:343] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:09.809103 30893 tablet_bootstrap.cc:654] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.808831 30662 leader_election.cc:304] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247; no voters: 39d7bd8c4f38482b8548151b5f717bcf, 4598d92c93d34ad5b95e906a309af48e
I20250831 07:57:09.810154 30897 raft_consensus.cc:2747] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:57:09.812069 30897 raft_consensus.cc:491] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:09.812506 30897 raft_consensus.cc:513] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.813550 30892 raft_consensus.cc:357] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.815076 30896 raft_consensus.cc:491] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:09.816121 30774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fffdf93425de4d52b8d33fd7cb7ecb9a" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf" is_pre_election: true
I20250831 07:57:09.816133 30897 leader_election.cc:290] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:09.815528 30896 raft_consensus.cc:513] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.817756 30891 tablet_bootstrap.cc:654] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.817934 30848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fffdf93425de4d52b8d33fd7cb7ecb9a" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4598d92c93d34ad5b95e906a309af48e" is_pre_election: true
I20250831 07:57:09.818351 30896 leader_election.cc:290] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:09.814213 30892 raft_consensus.cc:383] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.819228 30892 raft_consensus.cc:738] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.821010 30698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a48a1d49224a53a0949ef880502472" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247" is_pre_election: true
W20250831 07:57:09.821399 30665 leader_election.cc:343] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:09.821563 30699 raft_consensus.cc:2466] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 1.
I20250831 07:57:09.821722 30698 raft_consensus.cc:2466] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 0.
I20250831 07:57:09.822299 30848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a48a1d49224a53a0949ef880502472" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4598d92c93d34ad5b95e906a309af48e" is_pre_election: true
I20250831 07:57:09.822953 30848 raft_consensus.cc:2466] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 0.
I20250831 07:57:09.823428 30738 leader_election.cc:304] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 39d7bd8c4f38482b8548151b5f717bcf; no voters: 4598d92c93d34ad5b95e906a309af48e
I20250831 07:57:09.824477 30740 leader_election.cc:304] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d7bd8c4f38482b8548151b5f717bcf, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:09.825390 30896 raft_consensus.cc:2802] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:09.822999 30892 consensus_queue.cc:260] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.827560 30774 raft_consensus.cc:2466] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 0.
I20250831 07:57:09.834431 30921 raft_consensus.cc:2802] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:09.834774 30921 raft_consensus.cc:491] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:09.835032 30921 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:09.829861 30917 raft_consensus.cc:513] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.842125 30917 leader_election.cc:290] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:09.844142 30698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8524e0bbc5a04120a1862c3f2b492c3f" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:09.851497 30662 leader_election.cc:304] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 39d7bd8c4f38482b8548151b5f717bcf; no voters: 4598d92c93d34ad5b95e906a309af48e
I20250831 07:57:09.853291 30897 raft_consensus.cc:2802] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:09.853657 30897 raft_consensus.cc:491] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:57:09.853888 30738 leader_election.cc:343] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:09.854033 30897 raft_consensus.cc:3058] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:09.856911 30848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8524e0bbc5a04120a1862c3f2b492c3f" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:09.854713 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247" is_pre_election: true
I20250831 07:57:09.857530 30848 raft_consensus.cc:3058] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:57:09.858752 30812 leader_election.cc:343] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:09.870183 30774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf" is_pre_election: true
I20250831 07:57:09.870862 30774 raft_consensus.cc:2466] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 0.
I20250831 07:57:09.872864 30811 leader_election.cc:304] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39d7bd8c4f38482b8548151b5f717bcf, 4598d92c93d34ad5b95e906a309af48e; no voters: 357750ee400f42f8a09a88487e8bd247
I20250831 07:57:09.873728 30901 raft_consensus.cc:2802] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:09.874094 30901 raft_consensus.cc:491] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:09.874401 30901 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:09.861322 30921 raft_consensus.cc:513] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.872603 30848 raft_consensus.cc:2466] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 1.
I20250831 07:57:09.872768 30896 raft_consensus.cc:695] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [term 1 LEADER]: Becoming Leader. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:09.881281 30897 raft_consensus.cc:513] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.883492 30897 leader_election.cc:290] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 election: Requested vote from peers 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:09.884068 30774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fffdf93425de4d52b8d33fd7cb7ecb9a" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:09.884969 30896 consensus_queue.cc:237] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.888172 30774 raft_consensus.cc:3058] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:09.885988 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fffdf93425de4d52b8d33fd7cb7ecb9a" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:09.890089 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a48a1d49224a53a0949ef880502472" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:09.888206 30740 leader_election.cc:304] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39d7bd8c4f38482b8548151b5f717bcf, 4598d92c93d34ad5b95e906a309af48e; no voters: 357750ee400f42f8a09a88487e8bd247
I20250831 07:57:09.893946 30922 raft_consensus.cc:2802] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:09.886993 30901 raft_consensus.cc:513] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.894467 30922 raft_consensus.cc:695] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 1 LEADER]: Becoming Leader. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:09.894785 30891 tablet_bootstrap.cc:492] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247: No bootstrap required, opened a new log
W20250831 07:57:09.895200 30665 leader_election.cc:343] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:09.896772 30891 ts_tablet_manager.cc:1397] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247: Time spent bootstrapping tablet: real 0.104s user 0.021s sys 0.021s
I20250831 07:57:09.895306 30699 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:09.891085 30921 leader_election.cc:290] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:09.896992 30922 consensus_queue.cc:237] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.899466 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a48a1d49224a53a0949ef880502472" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:09.900053 30847 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:09.900735 30893 tablet_bootstrap.cc:492] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e: No bootstrap required, opened a new log
I20250831 07:57:09.902830 30893 ts_tablet_manager.cc:1397] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e: Time spent bootstrapping tablet: real 0.157s user 0.022s sys 0.014s
I20250831 07:57:09.903522 30892 ts_tablet_manager.cc:1428] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf: Time spent starting tablet: real 0.097s user 0.002s sys 0.015s
I20250831 07:57:09.903939 30698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:09.904839 30892 tablet_bootstrap.cc:492] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf: Bootstrap starting.
W20250831 07:57:09.905335 30812 leader_election.cc:343] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:09.907313 30901 leader_election.cc:290] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:09.911265 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:09.912694 30773 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:09.921622 30699 raft_consensus.cc:2466] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 1.
I20250831 07:57:09.923250 30738 leader_election.cc:304] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 39d7bd8c4f38482b8548151b5f717bcf; no voters:
I20250831 07:57:09.925829 30925 raft_consensus.cc:2802] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:09.926766 30925 raft_consensus.cc:695] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 LEADER]: Becoming Leader. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:09.927939 30925 consensus_queue.cc:237] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.932737 30774 raft_consensus.cc:2466] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 1.
I20250831 07:57:09.934111 30893 raft_consensus.cc:357] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.933897 30773 raft_consensus.cc:2466] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 1.
I20250831 07:57:09.934926 30662 leader_election.cc:304] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 39d7bd8c4f38482b8548151b5f717bcf; no voters: 4598d92c93d34ad5b95e906a309af48e
I20250831 07:57:09.934899 30893 raft_consensus.cc:383] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.935958 30847 raft_consensus.cc:2466] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 1.
I20250831 07:57:09.937549 30897 raft_consensus.cc:2802] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:09.936769 30893 raft_consensus.cc:738] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.939657 30811 leader_election.cc:304] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39d7bd8c4f38482b8548151b5f717bcf, 4598d92c93d34ad5b95e906a309af48e; no voters: 357750ee400f42f8a09a88487e8bd247
I20250831 07:57:09.941442 30901 raft_consensus.cc:2802] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:09.954871 30892 tablet_bootstrap.cc:654] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:09.955665 30596 catalog_manager.cc:5582] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2). New cstate: current_term: 1 leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:09.956665 30891 raft_consensus.cc:357] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.957572 30891 raft_consensus.cc:383] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:09.957964 30891 raft_consensus.cc:738] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Initialized, Role: FOLLOWER
I20250831 07:57:09.958860 30891 consensus_queue.cc:260] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.948442 30893 consensus_queue.cc:260] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:09.986332 30896 raft_consensus.cc:491] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:09.987049 30896 raft_consensus.cc:513] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.012370 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247" is_pre_election: true
I20250831 07:57:09.969152 30901 raft_consensus.cc:695] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Becoming Leader. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
W20250831 07:57:10.014143 30738 leader_election.cc:343] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:10.015232 30896 leader_election.cc:290] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:10.016467 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4598d92c93d34ad5b95e906a309af48e" is_pre_election: true
I20250831 07:57:10.043394 30893 ts_tablet_manager.cc:1428] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e: Time spent starting tablet: real 0.140s user 0.006s sys 0.008s
I20250831 07:57:10.046849 30891 ts_tablet_manager.cc:1428] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247: Time spent starting tablet: real 0.150s user 0.006s sys 0.034s
I20250831 07:57:10.047582 30893 tablet_bootstrap.cc:492] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e: Bootstrap starting.
W20250831 07:57:10.047775 30740 leader_election.cc:343] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:10.048013 30891 tablet_bootstrap.cc:492] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247: Bootstrap starting.
I20250831 07:57:10.048246 30740 leader_election.cc:304] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39d7bd8c4f38482b8548151b5f717bcf; no voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e
I20250831 07:57:10.049237 30896 raft_consensus.cc:2747] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:57:10.038547 30933 raft_consensus.cc:491] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:10.053630 30933 raft_consensus.cc:513] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.057104 30891 tablet_bootstrap.cc:654] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:10.057278 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27b19174fafc4d6bb0f98dc93d13dda4" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247" is_pre_election: true
I20250831 07:57:10.057978 30699 raft_consensus.cc:2466] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 0.
I20250831 07:57:10.059100 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27b19174fafc4d6bb0f98dc93d13dda4" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf" is_pre_election: true
I20250831 07:57:10.014369 30901 consensus_queue.cc:237] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.060586 30812 leader_election.cc:304] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:10.061542 30934 raft_consensus.cc:2802] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:10.061975 30934 raft_consensus.cc:491] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:10.062003 30933 leader_election.cc:290] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:10.062402 30934 raft_consensus.cc:3058] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:57:10.062680 30811 leader_election.cc:343] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:10.063952 30897 raft_consensus.cc:695] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 1 LEADER]: Becoming Leader. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Running, Role: LEADER
I20250831 07:57:10.064908 30897 consensus_queue.cc:237] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.075735 30892 tablet_bootstrap.cc:492] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf: No bootstrap required, opened a new log
I20250831 07:57:10.076239 30892 ts_tablet_manager.cc:1397] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf: Time spent bootstrapping tablet: real 0.172s user 0.012s sys 0.048s
I20250831 07:57:10.078055 30596 catalog_manager.cc:5582] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2). New cstate: current_term: 1 leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:10.070261 30934 raft_consensus.cc:513] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.079895 30596 catalog_manager.cc:5582] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2). New cstate: current_term: 1 leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:10.081910 30934 leader_election.cc:290] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:10.083266 30892 raft_consensus.cc:357] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.083987 30892 raft_consensus.cc:383] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:10.084079 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27b19174fafc4d6bb0f98dc93d13dda4" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:10.084309 30892 raft_consensus.cc:738] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Initialized, Role: FOLLOWER
I20250831 07:57:10.084640 30699 raft_consensus.cc:3058] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.084998 30892 consensus_queue.cc:260] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.083719 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27b19174fafc4d6bb0f98dc93d13dda4" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:10.086999 30773 raft_consensus.cc:3058] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.096769 30893 tablet_bootstrap.cc:654] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:10.105585 30891 tablet_bootstrap.cc:492] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247: No bootstrap required, opened a new log
I20250831 07:57:10.106101 30891 ts_tablet_manager.cc:1397] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247: Time spent bootstrapping tablet: real 0.058s user 0.013s sys 0.009s
I20250831 07:57:10.106611 30699 raft_consensus.cc:2466] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 1.
I20250831 07:57:10.107712 30812 leader_election.cc:304] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:10.108620 30901 raft_consensus.cc:2802] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:10.109325 30901 raft_consensus.cc:695] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Becoming Leader. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:10.109629 30892 ts_tablet_manager.cc:1428] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20250831 07:57:10.110603 30892 tablet_bootstrap.cc:492] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf: Bootstrap starting.
I20250831 07:57:10.110213 30901 consensus_queue.cc:237] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.113730 30773 raft_consensus.cc:2466] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 1.
I20250831 07:57:10.122860 30892 tablet_bootstrap.cc:654] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:10.123559 30594 catalog_manager.cc:5582] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 reported cstate change: term changed from 0 to 1, leader changed from <none> to 357750ee400f42f8a09a88487e8bd247 (127.29.217.1). New cstate: current_term: 1 leader_uuid: "357750ee400f42f8a09a88487e8bd247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:10.129369 30893 tablet_bootstrap.cc:492] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e: No bootstrap required, opened a new log
I20250831 07:57:10.130141 30893 ts_tablet_manager.cc:1397] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e: Time spent bootstrapping tablet: real 0.083s user 0.014s sys 0.003s
I20250831 07:57:10.133301 30893 raft_consensus.cc:357] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.134006 30893 raft_consensus.cc:383] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:10.134289 30893 raft_consensus.cc:738] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Initialized, Role: FOLLOWER
I20250831 07:57:10.134989 30891 raft_consensus.cc:357] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.136477 30891 raft_consensus.cc:383] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:10.136914 30891 raft_consensus.cc:738] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Initialized, Role: FOLLOWER
I20250831 07:57:10.134986 30893 consensus_queue.cc:260] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.137915 30891 consensus_queue.cc:260] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.139024 30893 ts_tablet_manager.cc:1428] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250831 07:57:10.139966 30893 tablet_bootstrap.cc:492] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e: Bootstrap starting.
I20250831 07:57:10.141255 30891 ts_tablet_manager.cc:1428] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247: Time spent starting tablet: real 0.035s user 0.002s sys 0.007s
I20250831 07:57:10.141057 30595 catalog_manager.cc:5582] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4598d92c93d34ad5b95e906a309af48e (127.29.217.3). New cstate: current_term: 1 leader_uuid: "4598d92c93d34ad5b95e906a309af48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:10.142289 30891 tablet_bootstrap.cc:492] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247: Bootstrap starting.
I20250831 07:57:10.148696 30893 tablet_bootstrap.cc:654] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:10.153589 30891 tablet_bootstrap.cc:654] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:10.165832 30893 tablet_bootstrap.cc:492] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e: No bootstrap required, opened a new log
I20250831 07:57:10.166244 30893 ts_tablet_manager.cc:1397] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.003s
I20250831 07:57:10.170179 30892 tablet_bootstrap.cc:492] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf: No bootstrap required, opened a new log
I20250831 07:57:10.170630 30892 ts_tablet_manager.cc:1397] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.013s
I20250831 07:57:10.171000 30891 tablet_bootstrap.cc:492] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247: No bootstrap required, opened a new log
I20250831 07:57:10.171443 30891 ts_tablet_manager.cc:1397] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.016s
I20250831 07:57:10.172807 30595 catalog_manager.cc:5582] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4598d92c93d34ad5b95e906a309af48e (127.29.217.3). New cstate: current_term: 1 leader_uuid: "4598d92c93d34ad5b95e906a309af48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:10.173594 30892 raft_consensus.cc:357] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.174201 30892 raft_consensus.cc:383] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:10.174463 30892 raft_consensus.cc:738] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Initialized, Role: FOLLOWER
I20250831 07:57:10.174911 30891 raft_consensus.cc:357] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.175159 30892 consensus_queue.cc:260] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.177464 30892 ts_tablet_manager.cc:1428] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250831 07:57:10.178306 30892 tablet_bootstrap.cc:492] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf: Bootstrap starting.
I20250831 07:57:10.175588 30891 raft_consensus.cc:383] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:10.181840 30891 raft_consensus.cc:738] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Initialized, Role: FOLLOWER
I20250831 07:57:10.182565 30891 consensus_queue.cc:260] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.183740 30892 tablet_bootstrap.cc:654] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:10.185654 30891 ts_tablet_manager.cc:1428] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20250831 07:57:10.186561 30891 tablet_bootstrap.cc:492] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247: Bootstrap starting.
I20250831 07:57:10.195413 30893 raft_consensus.cc:357] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.196173 30893 raft_consensus.cc:383] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:10.196357 30892 tablet_bootstrap.cc:492] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf: No bootstrap required, opened a new log
I20250831 07:57:10.196494 30893 raft_consensus.cc:738] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Initialized, Role: FOLLOWER
I20250831 07:57:10.196851 30892 ts_tablet_manager.cc:1397] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.007s
I20250831 07:57:10.198971 30892 raft_consensus.cc:357] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.199626 30892 raft_consensus.cc:383] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:10.199945 30892 raft_consensus.cc:738] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Initialized, Role: FOLLOWER
I20250831 07:57:10.200579 30892 consensus_queue.cc:260] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.202122 30891 tablet_bootstrap.cc:654] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:10.202486 30892 ts_tablet_manager.cc:1428] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250831 07:57:10.197243 30893 consensus_queue.cc:260] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.205358 30893 ts_tablet_manager.cc:1428] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e: Time spent starting tablet: real 0.039s user 0.008s sys 0.000s
I20250831 07:57:10.206331 30893 tablet_bootstrap.cc:492] T d0b161ad222348bea9cf99d0d2dc923d P 4598d92c93d34ad5b95e906a309af48e: Bootstrap starting.
I20250831 07:57:10.212833 30893 tablet_bootstrap.cc:654] T d0b161ad222348bea9cf99d0d2dc923d P 4598d92c93d34ad5b95e906a309af48e: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:10.217311 30891 tablet_bootstrap.cc:492] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247: No bootstrap required, opened a new log
I20250831 07:57:10.217777 30891 ts_tablet_manager.cc:1397] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.004s
I20250831 07:57:10.219955 30891 raft_consensus.cc:357] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.220580 30891 raft_consensus.cc:383] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:10.220835 30891 raft_consensus.cc:738] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Initialized, Role: FOLLOWER
I20250831 07:57:10.221661 30891 consensus_queue.cc:260] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.222393 30897 raft_consensus.cc:491] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:10.222828 30897 raft_consensus.cc:513] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.227792 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8524e0bbc5a04120a1862c3f2b492c3f" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf" is_pre_election: true
I20250831 07:57:10.227954 30897 leader_election.cc:290] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:10.230262 30891 ts_tablet_manager.cc:1428] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20250831 07:57:10.230191 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8524e0bbc5a04120a1862c3f2b492c3f" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4598d92c93d34ad5b95e906a309af48e" is_pre_election: true
I20250831 07:57:10.231294 30891 tablet_bootstrap.cc:492] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247: Bootstrap starting.
I20250831 07:57:10.231415 30847 raft_consensus.cc:2391] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 357750ee400f42f8a09a88487e8bd247 in current term 1: Already voted for candidate 39d7bd8c4f38482b8548151b5f717bcf in this term.
I20250831 07:57:10.232379 30901 raft_consensus.cc:491] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:10.232972 30901 raft_consensus.cc:513] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.237393 30891 tablet_bootstrap.cc:654] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:10.237867 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fffdf93425de4d52b8d33fd7cb7ecb9a" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247" is_pre_election: true
I20250831 07:57:10.241303 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fffdf93425de4d52b8d33fd7cb7ecb9a" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf" is_pre_election: true
I20250831 07:57:10.241436 30665 leader_election.cc:304] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247; no voters: 39d7bd8c4f38482b8548151b5f717bcf, 4598d92c93d34ad5b95e906a309af48e
I20250831 07:57:10.242080 30773 raft_consensus.cc:2391] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4598d92c93d34ad5b95e906a309af48e in current term 1: Already voted for candidate 357750ee400f42f8a09a88487e8bd247 in this term.
I20250831 07:57:10.242199 30897 raft_consensus.cc:3058] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.242792 30891 tablet_bootstrap.cc:492] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247: No bootstrap required, opened a new log
I20250831 07:57:10.243206 30891 ts_tablet_manager.cc:1397] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20250831 07:57:10.243484 30811 leader_election.cc:304] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4598d92c93d34ad5b95e906a309af48e; no voters: 357750ee400f42f8a09a88487e8bd247, 39d7bd8c4f38482b8548151b5f717bcf
I20250831 07:57:10.235548 30901 leader_election.cc:290] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:10.244246 30934 raft_consensus.cc:3058] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.245675 30891 raft_consensus.cc:357] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.246331 30891 raft_consensus.cc:383] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:10.246592 30891 raft_consensus.cc:738] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Initialized, Role: FOLLOWER
I20250831 07:57:10.247237 30891 consensus_queue.cc:260] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.249276 30897 raft_consensus.cc:2747] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:57:10.249392 30891 ts_tablet_manager.cc:1428] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20250831 07:57:10.251184 30934 raft_consensus.cc:2747] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:57:10.251768 30893 tablet_bootstrap.cc:492] T d0b161ad222348bea9cf99d0d2dc923d P 4598d92c93d34ad5b95e906a309af48e: No bootstrap required, opened a new log
I20250831 07:57:10.252223 30893 ts_tablet_manager.cc:1397] T d0b161ad222348bea9cf99d0d2dc923d P 4598d92c93d34ad5b95e906a309af48e: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.028s
I20250831 07:57:10.254179 30893 raft_consensus.cc:357] T d0b161ad222348bea9cf99d0d2dc923d P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.254601 30893 raft_consensus.cc:383] T d0b161ad222348bea9cf99d0d2dc923d P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:10.254804 30893 raft_consensus.cc:738] T d0b161ad222348bea9cf99d0d2dc923d P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Initialized, Role: FOLLOWER
I20250831 07:57:10.255282 30893 consensus_queue.cc:260] T d0b161ad222348bea9cf99d0d2dc923d P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.257174 30893 ts_tablet_manager.cc:1428] T d0b161ad222348bea9cf99d0d2dc923d P 4598d92c93d34ad5b95e906a309af48e: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20250831 07:57:10.259953 30893 tablet_bootstrap.cc:492] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e: Bootstrap starting.
I20250831 07:57:10.264601 30934 raft_consensus.cc:491] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:10.265512 30893 tablet_bootstrap.cc:654] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:10.265066 30934 raft_consensus.cc:513] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.267735 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247" is_pre_election: true
I20250831 07:57:10.268348 30699 raft_consensus.cc:2466] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 0.
I20250831 07:57:10.269341 30812 leader_election.cc:304] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:10.270134 30901 raft_consensus.cc:2802] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:10.270469 30901 raft_consensus.cc:491] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:10.270780 30901 raft_consensus.cc:3058] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.272450 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf" is_pre_election: true
I20250831 07:57:10.273000 30773 raft_consensus.cc:2466] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 0.
I20250831 07:57:10.273618 30934 leader_election.cc:290] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
W20250831 07:57:10.275228 30644 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 07:57:10.277210 30901 raft_consensus.cc:513] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.279036 30893 tablet_bootstrap.cc:492] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e: No bootstrap required, opened a new log
I20250831 07:57:10.279043 30901 leader_election.cc:290] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:10.279736 30893 ts_tablet_manager.cc:1397] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.004s
I20250831 07:57:10.280148 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:10.280573 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:10.280683 30699 raft_consensus.cc:3058] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.281131 30773 raft_consensus.cc:3058] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.282565 30893 raft_consensus.cc:357] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.283280 30893 raft_consensus.cc:383] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:10.283589 30893 raft_consensus.cc:738] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Initialized, Role: FOLLOWER
I20250831 07:57:10.284438 30893 consensus_queue.cc:260] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.287268 30893 ts_tablet_manager.cc:1428] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250831 07:57:10.287539 30699 raft_consensus.cc:2466] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 1.
I20250831 07:57:10.288798 30812 leader_election.cc:304] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:10.289574 30901 raft_consensus.cc:2802] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:10.290169 30901 raft_consensus.cc:695] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Becoming Leader. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:10.287501 30773 raft_consensus.cc:2466] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 1.
I20250831 07:57:10.291016 30901 consensus_queue.cc:237] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.300629 30595 catalog_manager.cc:5582] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4598d92c93d34ad5b95e906a309af48e (127.29.217.3). New cstate: current_term: 1 leader_uuid: "4598d92c93d34ad5b95e906a309af48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:10.321393 30896 consensus_queue.cc:1035] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.340808 30896 consensus_queue.cc:1035] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.350384 30896 consensus_queue.cc:1035] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.365067 30917 consensus_queue.cc:1035] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:57:10.376560 30847 raft_consensus.cc:3058] T d0b161ad222348bea9cf99d0d2dc923d P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.390730 30922 consensus_queue.cc:1035] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.392335 30925 consensus_queue.cc:1035] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.415800 30901 raft_consensus.cc:491] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:10.416334 30901 raft_consensus.cc:513] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.418233 30901 leader_election.cc:290] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:10.420207 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24dbfaab9a7a42ffa64a24fa6f384740" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247" is_pre_election: true
I20250831 07:57:10.420855 30699 raft_consensus.cc:2466] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 0.
I20250831 07:57:10.422515 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24dbfaab9a7a42ffa64a24fa6f384740" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf" is_pre_election: true
I20250831 07:57:10.423202 30773 raft_consensus.cc:2466] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 0.
I20250831 07:57:10.424455 30812 leader_election.cc:304] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:10.425324 30901 raft_consensus.cc:2802] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:10.426342 30901 raft_consensus.cc:491] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:10.426749 30901 raft_consensus.cc:3058] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.432649 30901 raft_consensus.cc:513] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.434136 30901 leader_election.cc:290] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:10.434804 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24dbfaab9a7a42ffa64a24fa6f384740" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:10.435302 30699 raft_consensus.cc:3058] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.435225 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24dbfaab9a7a42ffa64a24fa6f384740" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:10.435848 30773 raft_consensus.cc:3058] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.440429 30699 raft_consensus.cc:2466] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 1.
I20250831 07:57:10.441350 30812 leader_election.cc:304] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:10.441967 30901 raft_consensus.cc:2802] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:10.442267 30773 raft_consensus.cc:2466] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 1.
I20250831 07:57:10.442335 30901 raft_consensus.cc:695] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Becoming Leader. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:10.443580 30901 consensus_queue.cc:237] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.452107 30595 catalog_manager.cc:5582] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4598d92c93d34ad5b95e906a309af48e (127.29.217.3). New cstate: current_term: 1 leader_uuid: "4598d92c93d34ad5b95e906a309af48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:10.525667 30901 consensus_queue.cc:1035] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.537590 30699 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.547935 30901 consensus_queue.cc:1035] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.552917 30897 consensus_queue.cc:1035] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.568470 30897 consensus_queue.cc:1035] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.617888 30928 raft_consensus.cc:491] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:10.618330 30928 raft_consensus.cc:513] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.619995 30928 leader_election.cc:290] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:10.620920 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf" is_pre_election: true
I20250831 07:57:10.621032 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4598d92c93d34ad5b95e906a309af48e" is_pre_election: true
I20250831 07:57:10.621646 30847 raft_consensus.cc:2466] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 0.
I20250831 07:57:10.622056 30934 raft_consensus.cc:491] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:10.622226 30773 raft_consensus.cc:2466] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 0.
I20250831 07:57:10.622514 30934 raft_consensus.cc:513] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.622799 30665 leader_election.cc:304] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:10.623705 30928 raft_consensus.cc:2802] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:10.624087 30928 raft_consensus.cc:491] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:10.624401 30928 raft_consensus.cc:3058] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.624449 30934 leader_election.cc:290] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:10.625833 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247" is_pre_election: true
I20250831 07:57:10.626048 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf" is_pre_election: true
I20250831 07:57:10.626555 30773 raft_consensus.cc:2466] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 0.
I20250831 07:57:10.627359 30811 leader_election.cc:304] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d7bd8c4f38482b8548151b5f717bcf, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:10.628006 30934 raft_consensus.cc:2802] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:10.628296 30934 raft_consensus.cc:491] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:10.628553 30934 raft_consensus.cc:3058] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.632929 30928 raft_consensus.cc:513] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.634342 30699 raft_consensus.cc:2391] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4598d92c93d34ad5b95e906a309af48e in current term 1: Already voted for candidate 357750ee400f42f8a09a88487e8bd247 in this term.
I20250831 07:57:10.635625 30928 leader_election.cc:290] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 election: Requested vote from peers 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:10.636300 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:10.636891 30773 raft_consensus.cc:3058] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:10.636974 30934 raft_consensus.cc:513] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.637538 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:10.638276 30847 raft_consensus.cc:2391] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 357750ee400f42f8a09a88487e8bd247 in current term 1: Already voted for candidate 4598d92c93d34ad5b95e906a309af48e in this term.
I20250831 07:57:10.639353 30934 leader_election.cc:290] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:10.640129 30774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:10.641361 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:10.642077 30699 raft_consensus.cc:2391] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4598d92c93d34ad5b95e906a309af48e in current term 1: Already voted for candidate 357750ee400f42f8a09a88487e8bd247 in this term.
I20250831 07:57:10.642972 30812 leader_election.cc:304] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4598d92c93d34ad5b95e906a309af48e; no voters: 357750ee400f42f8a09a88487e8bd247, 39d7bd8c4f38482b8548151b5f717bcf
I20250831 07:57:10.643735 30934 raft_consensus.cc:2747] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250831 07:57:10.644292 30773 raft_consensus.cc:2466] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 1.
I20250831 07:57:10.645174 30662 leader_election.cc:304] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 39d7bd8c4f38482b8548151b5f717bcf; no voters: 4598d92c93d34ad5b95e906a309af48e
I20250831 07:57:10.646288 30928 raft_consensus.cc:2802] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:10.646699 30928 raft_consensus.cc:695] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 1 LEADER]: Becoming Leader. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Running, Role: LEADER
I20250831 07:57:10.647424 30928 consensus_queue.cc:237] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:10.655189 30596 catalog_manager.cc:5582] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 reported cstate change: term changed from 0 to 1, leader changed from <none> to 357750ee400f42f8a09a88487e8bd247 (127.29.217.1). New cstate: current_term: 1 leader_uuid: "357750ee400f42f8a09a88487e8bd247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:10.692798 30934 consensus_queue.cc:1035] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.705237 30929 consensus_queue.cc:1035] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:57:10.724792 30929 consensus_queue.cc:1035] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.734928 30929 consensus_queue.cc:1035] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.929572 30929 consensus_queue.cc:1035] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.960901 30901 consensus_queue.cc:1035] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:10.973232 30847 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b"
dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
mode: GRACEFUL
new_leader_uuid: "357750ee400f42f8a09a88487e8bd247"
from {username='slave'} at 127.0.0.1:49182
I20250831 07:57:10.973920 30847 raft_consensus.cc:604] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Received request to transfer leadership to TS 357750ee400f42f8a09a88487e8bd247
I20250831 07:57:10.998769 30773 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "47a48a1d49224a53a0949ef880502472"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
mode: GRACEFUL
new_leader_uuid: "357750ee400f42f8a09a88487e8bd247"
from {username='slave'} at 127.0.0.1:43450
I20250831 07:57:10.999253 30773 raft_consensus.cc:604] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 LEADER]: Received request to transfer leadership to TS 357750ee400f42f8a09a88487e8bd247
I20250831 07:57:11.007113 30847 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "24dbfaab9a7a42ffa64a24fa6f384740"
dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
mode: GRACEFUL
new_leader_uuid: "357750ee400f42f8a09a88487e8bd247"
from {username='slave'} at 127.0.0.1:49182
I20250831 07:57:11.007615 30847 raft_consensus.cc:604] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Received request to transfer leadership to TS 357750ee400f42f8a09a88487e8bd247
I20250831 07:57:11.015659 30847 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "27b19174fafc4d6bb0f98dc93d13dda4"
dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
mode: GRACEFUL
new_leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:49182
I20250831 07:57:11.016198 30847 raft_consensus.cc:604] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Received request to transfer leadership to TS 39d7bd8c4f38482b8548151b5f717bcf
I20250831 07:57:11.023905 30847 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66"
dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
mode: GRACEFUL
new_leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:49182
I20250831 07:57:11.024508 30847 raft_consensus.cc:604] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Received request to transfer leadership to TS 39d7bd8c4f38482b8548151b5f717bcf
I20250831 07:57:11.039016 30699 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f"
dest_uuid: "357750ee400f42f8a09a88487e8bd247"
mode: GRACEFUL
new_leader_uuid: "4598d92c93d34ad5b95e906a309af48e"
from {username='slave'} at 127.0.0.1:37486
I20250831 07:57:11.039626 30699 raft_consensus.cc:604] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 1 LEADER]: Received request to transfer leadership to TS 4598d92c93d34ad5b95e906a309af48e
I20250831 07:57:11.082988 30928 consensus_queue.cc:1035] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:57:11.094681 30928 consensus_queue.cc:1035] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:11.104032 30897 raft_consensus.cc:991] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247: : Instructing follower 4598d92c93d34ad5b95e906a309af48e to start an election
I20250831 07:57:11.104485 30928 raft_consensus.cc:1079] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 1 LEADER]: Signalling peer 4598d92c93d34ad5b95e906a309af48e to start an election
I20250831 07:57:11.106127 30847 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f"
dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
from {username='slave'} at 127.0.0.1:49176
I20250831 07:57:11.106720 30847 raft_consensus.cc:491] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:11.107095 30847 raft_consensus.cc:3058] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.116277 30847 raft_consensus.cc:513] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.118713 30847 leader_election.cc:290] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 2 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:11.119112 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:11.119657 30699 raft_consensus.cc:3053] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:11.120008 30699 raft_consensus.cc:738] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Running, Role: LEADER
I20250831 07:57:11.120731 30699 consensus_queue.cc:260] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.121220 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68688b732e6a47858fa4d8257adf011f" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:11.121783 30773 raft_consensus.cc:3058] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.122072 30699 raft_consensus.cc:3058] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.127050 30773 raft_consensus.cc:2466] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 2.
I20250831 07:57:11.128327 30699 raft_consensus.cc:2466] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 2.
I20250831 07:57:11.128518 30811 leader_election.cc:304] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d7bd8c4f38482b8548151b5f717bcf, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:11.129217 30901 raft_consensus.cc:2802] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:11.129693 30901 raft_consensus.cc:695] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 2 LEADER]: Becoming Leader. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:11.130451 30901 consensus_queue.cc:237] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.137174 30595 catalog_manager.cc:5582] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e reported cstate change: term changed from 1 to 2, leader changed from 357750ee400f42f8a09a88487e8bd247 (127.29.217.1) to 4598d92c93d34ad5b95e906a309af48e (127.29.217.3). New cstate: current_term: 2 leader_uuid: "4598d92c93d34ad5b95e906a309af48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:11.155961 30929 raft_consensus.cc:991] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e: : Instructing follower 357750ee400f42f8a09a88487e8bd247 to start an election
I20250831 07:57:11.156431 30934 raft_consensus.cc:1079] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Signalling peer 357750ee400f42f8a09a88487e8bd247 to start an election
I20250831 07:57:11.158005 30699 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b"
dest_uuid: "357750ee400f42f8a09a88487e8bd247"
from {username='slave'} at 127.0.0.1:37472
I20250831 07:57:11.158502 30699 raft_consensus.cc:491] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:11.158775 30699 raft_consensus.cc:3058] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.163058 30699 raft_consensus.cc:513] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.164531 30699 leader_election.cc:290] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 2 election: Requested vote from peers 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:11.165717 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:11.165922 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:11.166227 30847 raft_consensus.cc:3053] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:11.166461 30773 raft_consensus.cc:3058] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.166556 30847 raft_consensus.cc:738] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:11.167285 30847 consensus_queue.cc:260] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.168176 30847 raft_consensus.cc:3058] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.176532 30847 raft_consensus.cc:2466] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 2.
I20250831 07:57:11.176597 30773 raft_consensus.cc:2466] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 2.
I20250831 07:57:11.177505 30665 leader_election.cc:304] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:11.178263 30928 raft_consensus.cc:2802] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:11.178711 30928 raft_consensus.cc:695] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Becoming Leader. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Running, Role: LEADER
I20250831 07:57:11.179488 30928 consensus_queue.cc:237] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.186506 30595 catalog_manager.cc:5582] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 reported cstate change: term changed from 1 to 2, leader changed from 4598d92c93d34ad5b95e906a309af48e (127.29.217.3) to 357750ee400f42f8a09a88487e8bd247 (127.29.217.1). New cstate: current_term: 2 leader_uuid: "357750ee400f42f8a09a88487e8bd247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:11.357699 30934 raft_consensus.cc:991] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e: : Instructing follower 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:11.358207 30901 raft_consensus.cc:1079] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Signalling peer 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:11.359601 30773 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "27b19174fafc4d6bb0f98dc93d13dda4"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:43448
I20250831 07:57:11.360189 30773 raft_consensus.cc:491] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:11.360540 30773 raft_consensus.cc:3058] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.364482 30929 raft_consensus.cc:991] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e: : Instructing follower 357750ee400f42f8a09a88487e8bd247 to start an election
I20250831 07:57:11.364938 30929 raft_consensus.cc:1079] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Signalling peer 357750ee400f42f8a09a88487e8bd247 to start an election
I20250831 07:57:11.366474 30699 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "24dbfaab9a7a42ffa64a24fa6f384740"
dest_uuid: "357750ee400f42f8a09a88487e8bd247"
from {username='slave'} at 127.0.0.1:37472
I20250831 07:57:11.367187 30699 raft_consensus.cc:491] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:11.367637 30699 raft_consensus.cc:3058] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.369536 30773 raft_consensus.cc:513] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.373116 30698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27b19174fafc4d6bb0f98dc93d13dda4" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:11.373217 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27b19174fafc4d6bb0f98dc93d13dda4" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:11.374014 30698 raft_consensus.cc:3058] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.374363 30847 raft_consensus.cc:3053] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:11.376601 30847 raft_consensus.cc:738] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:11.376259 30699 raft_consensus.cc:513] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.377629 30847 consensus_queue.cc:260] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.379202 30847 raft_consensus.cc:3058] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.379884 30699 leader_election.cc:290] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 2 election: Requested vote from peers 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:11.379841 30774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24dbfaab9a7a42ffa64a24fa6f384740" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:11.380548 30774 raft_consensus.cc:3058] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.384861 30846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24dbfaab9a7a42ffa64a24fa6f384740" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:11.385725 30846 raft_consensus.cc:3053] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:11.386165 30846 raft_consensus.cc:738] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:11.373874 30773 leader_election.cc:290] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 2 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:11.387216 30846 consensus_queue.cc:260] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.393725 30698 raft_consensus.cc:2466] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 2.
I20250831 07:57:11.394459 30846 raft_consensus.cc:3058] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.394935 30738 leader_election.cc:304] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 39d7bd8c4f38482b8548151b5f717bcf; no voters:
I20250831 07:57:11.396471 30896 raft_consensus.cc:2802] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:11.396936 30896 raft_consensus.cc:695] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 LEADER]: Becoming Leader. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:11.397720 30896 consensus_queue.cc:237] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.401046 30846 raft_consensus.cc:2466] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 2.
I20250831 07:57:11.402213 30665 leader_election.cc:304] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:11.404050 30928 raft_consensus.cc:2802] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:11.404196 30847 raft_consensus.cc:2466] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 2.
I20250831 07:57:11.404495 30928 raft_consensus.cc:695] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Becoming Leader. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Running, Role: LEADER
I20250831 07:57:11.405277 30928 consensus_queue.cc:237] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.406672 30595 catalog_manager.cc:5582] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf reported cstate change: term changed from 1 to 2, leader changed from 4598d92c93d34ad5b95e906a309af48e (127.29.217.3) to 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2). New cstate: current_term: 2 leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:11.408010 30774 raft_consensus.cc:2466] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 2.
I20250831 07:57:11.413713 30594 catalog_manager.cc:5582] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 reported cstate change: term changed from 1 to 2, leader changed from 4598d92c93d34ad5b95e906a309af48e (127.29.217.3) to 357750ee400f42f8a09a88487e8bd247 (127.29.217.1). New cstate: current_term: 2 leader_uuid: "357750ee400f42f8a09a88487e8bd247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:11.431236 30925 raft_consensus.cc:991] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf: : Instructing follower 357750ee400f42f8a09a88487e8bd247 to start an election
I20250831 07:57:11.431625 30896 raft_consensus.cc:1079] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 LEADER]: Signalling peer 357750ee400f42f8a09a88487e8bd247 to start an election
I20250831 07:57:11.432866 30698 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "47a48a1d49224a53a0949ef880502472"
dest_uuid: "357750ee400f42f8a09a88487e8bd247"
from {username='slave'} at 127.0.0.1:37468
I20250831 07:57:11.433281 30698 raft_consensus.cc:491] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:11.433563 30698 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.437858 30698 raft_consensus.cc:513] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.439296 30698 leader_election.cc:290] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 2 election: Requested vote from peers 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:11.440140 30774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a48a1d49224a53a0949ef880502472" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:11.440433 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a48a1d49224a53a0949ef880502472" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:11.440671 30774 raft_consensus.cc:3053] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:11.440927 30847 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.440973 30774 raft_consensus.cc:738] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:11.441735 30774 consensus_queue.cc:260] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.442636 30774 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.445885 30847 raft_consensus.cc:2466] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 2.
I20250831 07:57:11.446929 30665 leader_election.cc:304] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:11.447593 30929 raft_consensus.cc:991] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e: : Instructing follower 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:11.447635 30928 raft_consensus.cc:2802] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:11.448163 30901 raft_consensus.cc:1079] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Signalling peer 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:11.448213 30928 raft_consensus.cc:695] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Becoming Leader. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Running, Role: LEADER
I20250831 07:57:11.448969 30928 consensus_queue.cc:237] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.449573 30773 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:43448
I20250831 07:57:11.450012 30773 raft_consensus.cc:491] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:11.450325 30773 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.451570 30774 raft_consensus.cc:2466] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 2.
I20250831 07:57:11.456560 30594 catalog_manager.cc:5582] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 reported cstate change: term changed from 1 to 2, leader changed from 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2) to 357750ee400f42f8a09a88487e8bd247 (127.29.217.1). New cstate: current_term: 2 leader_uuid: "357750ee400f42f8a09a88487e8bd247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:11.457170 30773 raft_consensus.cc:513] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.459386 30773 leader_election.cc:290] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 2 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:11.459635 30698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:11.460008 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:11.460244 30698 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.460562 30847 raft_consensus.cc:3053] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:11.460875 30847 raft_consensus.cc:738] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:11.461642 30847 consensus_queue.cc:260] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.462790 30847 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:11.469544 30847 raft_consensus.cc:2466] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 2.
I20250831 07:57:11.470233 30698 raft_consensus.cc:2466] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 2.
I20250831 07:57:11.470512 30740 leader_election.cc:304] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d7bd8c4f38482b8548151b5f717bcf, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:11.471253 30896 raft_consensus.cc:2802] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:11.471685 30896 raft_consensus.cc:695] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 LEADER]: Becoming Leader. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:11.472440 30896 consensus_queue.cc:237] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:11.478109 30594 catalog_manager.cc:5582] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf reported cstate change: term changed from 1 to 2, leader changed from 4598d92c93d34ad5b95e906a309af48e (127.29.217.3) to 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2). New cstate: current_term: 2 leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:11.705808 30773 raft_consensus.cc:1273] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Refusing update from remote peer 357750ee400f42f8a09a88487e8bd247: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:11.706871 30928 consensus_queue.cc:1035] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:11.715238 30847 raft_consensus.cc:1273] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Refusing update from remote peer 357750ee400f42f8a09a88487e8bd247: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:11.716791 30985 consensus_queue.cc:1035] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:57:11.723121 30698 raft_consensus.cc:1273] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Refusing update from remote peer 4598d92c93d34ad5b95e906a309af48e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:11.724825 30929 consensus_queue.cc:1035] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:11.731940 30773 raft_consensus.cc:1273] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Refusing update from remote peer 4598d92c93d34ad5b95e906a309af48e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:11.733197 30901 consensus_queue.cc:1035] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:11.879499 30698 raft_consensus.cc:1273] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:11.880788 30896 consensus_queue.cc:1035] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:11.888561 30847 raft_consensus.cc:1273] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:11.889835 30896 consensus_queue.cc:1035] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:11.943686 30847 raft_consensus.cc:1273] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Refusing update from remote peer 357750ee400f42f8a09a88487e8bd247: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:11.945060 30985 consensus_queue.cc:1035] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:11.958004 30773 raft_consensus.cc:1273] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Refusing update from remote peer 357750ee400f42f8a09a88487e8bd247: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:11.959179 30992 consensus_queue.cc:1035] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:12.021045 30773 raft_consensus.cc:1273] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Refusing update from remote peer 357750ee400f42f8a09a88487e8bd247: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:12.022249 30992 consensus_queue.cc:1035] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:12.030721 30847 raft_consensus.cc:1273] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Refusing update from remote peer 357750ee400f42f8a09a88487e8bd247: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:12.032349 30928 consensus_queue.cc:1035] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:12.045619 30698 raft_consensus.cc:1273] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:12.046926 30896 consensus_queue.cc:1035] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:12.054889 30847 raft_consensus.cc:1273] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:12.057051 30922 consensus_queue.cc:1035] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:57:13.416075 30720 debug-util.cc:398] Leaking SignalData structure 0x7b08000e4060 after lost signal to thread 30567
W20250831 07:57:13.416781 30720 debug-util.cc:398] Leaking SignalData structure 0x7b08000e4740 after lost signal to thread 30630
W20250831 07:57:13.417469 30720 debug-util.cc:398] Leaking SignalData structure 0x7b08000b9820 after lost signal to thread 30723
W20250831 07:57:13.418177 30720 debug-util.cc:398] Leaking SignalData structure 0x7b08001b44c0 after lost signal to thread 30798
W20250831 07:57:13.418798 30720 debug-util.cc:398] Leaking SignalData structure 0x7b08001eebc0 after lost signal to thread 30872
I20250831 07:57:14.114758 30564 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20250831 07:57:14.115481 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:14.138157 30698 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "47a48a1d49224a53a0949ef880502472"
dest_uuid: "357750ee400f42f8a09a88487e8bd247"
mode: GRACEFUL
new_leader_uuid: "4598d92c93d34ad5b95e906a309af48e"
from {username='slave'} at 127.0.0.1:37490
I20250831 07:57:14.138651 30698 raft_consensus.cc:604] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Received request to transfer leadership to TS 4598d92c93d34ad5b95e906a309af48e
I20250831 07:57:14.151769 30773 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
mode: GRACEFUL
new_leader_uuid: "4598d92c93d34ad5b95e906a309af48e"
from {username='slave'} at 127.0.0.1:43462
I20250831 07:57:14.152518 30773 raft_consensus.cc:604] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 LEADER]: Received request to transfer leadership to TS 4598d92c93d34ad5b95e906a309af48e
I20250831 07:57:14.153956 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250831 07:57:14.154579 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:14.423720 30998 raft_consensus.cc:991] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf: : Instructing follower 4598d92c93d34ad5b95e906a309af48e to start an election
I20250831 07:57:14.424185 30925 raft_consensus.cc:1079] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 LEADER]: Signalling peer 4598d92c93d34ad5b95e906a309af48e to start an election
I20250831 07:57:14.425577 30848 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66"
dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
from {username='slave'} at 127.0.0.1:49170
I20250831 07:57:14.426167 30848 raft_consensus.cc:491] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:14.426540 30848 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:14.434595 30848 raft_consensus.cc:513] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:14.436760 30848 leader_election.cc:290] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 3 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:14.437669 30698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:14.437913 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:14.438264 30698 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:14.438410 30773 raft_consensus.cc:3053] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:57:14.438742 30773 raft_consensus.cc:738] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:14.439406 30773 consensus_queue.cc:260] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:14.440514 30773 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:14.446621 30698 raft_consensus.cc:2466] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 3.
I20250831 07:57:14.447461 30773 raft_consensus.cc:2466] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 3.
I20250831 07:57:14.447959 30812 leader_election.cc:304] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:14.448958 30901 raft_consensus.cc:2802] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:57:14.450314 30901 raft_consensus.cc:695] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 3 LEADER]: Becoming Leader. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:14.451332 30901 consensus_queue.cc:237] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:14.460292 30594 catalog_manager.cc:5582] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e reported cstate change: term changed from 2 to 3, leader changed from 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2) to 4598d92c93d34ad5b95e906a309af48e (127.29.217.3). New cstate: current_term: 3 leader_uuid: "4598d92c93d34ad5b95e906a309af48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:14.484512 31008 raft_consensus.cc:991] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247: : Instructing follower 4598d92c93d34ad5b95e906a309af48e to start an election
I20250831 07:57:14.485695 30985 raft_consensus.cc:1079] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Signalling peer 4598d92c93d34ad5b95e906a309af48e to start an election
I20250831 07:57:14.487934 30848 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "47a48a1d49224a53a0949ef880502472"
dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
from {username='slave'} at 127.0.0.1:49176
I20250831 07:57:14.488561 30848 raft_consensus.cc:491] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:14.488984 30848 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:14.496486 30848 raft_consensus.cc:513] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:14.499724 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a48a1d49224a53a0949ef880502472" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:14.500321 30773 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:14.500523 30698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a48a1d49224a53a0949ef880502472" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:14.501044 30698 raft_consensus.cc:3053] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:57:14.501353 30698 raft_consensus.cc:738] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Running, Role: LEADER
I20250831 07:57:14.502640 30698 consensus_queue.cc:260] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:14.503991 30698 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:14.507133 30773 raft_consensus.cc:2466] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 3.
I20250831 07:57:14.507982 30848 leader_election.cc:290] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 3 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:14.508458 30811 leader_election.cc:304] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d7bd8c4f38482b8548151b5f717bcf, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:14.509335 30901 raft_consensus.cc:2802] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:57:14.509805 30901 raft_consensus.cc:695] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 3 LEADER]: Becoming Leader. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:14.510576 30901 consensus_queue.cc:237] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:14.511511 30698 raft_consensus.cc:2466] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 3.
I20250831 07:57:14.517956 30594 catalog_manager.cc:5582] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e reported cstate change: term changed from 2 to 3, leader changed from 357750ee400f42f8a09a88487e8bd247 (127.29.217.1) to 4598d92c93d34ad5b95e906a309af48e (127.29.217.3). New cstate: current_term: 3 leader_uuid: "4598d92c93d34ad5b95e906a309af48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:14.908869 30698 raft_consensus.cc:1273] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Refusing update from remote peer 4598d92c93d34ad5b95e906a309af48e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:14.909972 30901 consensus_queue.cc:1035] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:14.922258 30773 raft_consensus.cc:1273] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Refusing update from remote peer 4598d92c93d34ad5b95e906a309af48e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:14.923730 30774 raft_consensus.cc:1273] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Refusing update from remote peer 4598d92c93d34ad5b95e906a309af48e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:14.924474 31013 consensus_queue.cc:1035] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:57:14.925691 30901 consensus_queue.cc:1035] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:14.941799 30698 raft_consensus.cc:1273] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Refusing update from remote peer 4598d92c93d34ad5b95e906a309af48e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:14.946461 30901 consensus_queue.cc:1035] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:15.155537 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:15.160290 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:15.160665 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:16.161538 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:16.166335 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:16.166740 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:17.167901 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:17.173290 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:17.173707 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:18.174705 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:18.179332 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:18.179721 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:19.180616 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:19.185300 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:19.185727 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:20.186800 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:20.192663 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:20.193120 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:21.194070 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:21.198796 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:21.199198 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:22.200112 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:22.205355 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:22.205825 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:23.206769 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:23.211493 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:23.211916 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:24.238826 30564 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20250831 07:57:24.242986 30698 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b"
dest_uuid: "357750ee400f42f8a09a88487e8bd247"
mode: GRACEFUL
new_leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:37486
I20250831 07:57:24.243597 30698 raft_consensus.cc:604] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Received request to transfer leadership to TS 39d7bd8c4f38482b8548151b5f717bcf
I20250831 07:57:24.248347 30847 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "47a48a1d49224a53a0949ef880502472"
dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
mode: GRACEFUL
new_leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:49182
I20250831 07:57:24.248823 30847 raft_consensus.cc:604] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 3 LEADER]: Received request to transfer leadership to TS 39d7bd8c4f38482b8548151b5f717bcf
I20250831 07:57:24.253073 30698 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "fffdf93425de4d52b8d33fd7cb7ecb9a"
dest_uuid: "357750ee400f42f8a09a88487e8bd247"
mode: GRACEFUL
new_leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:37486
I20250831 07:57:24.253561 30698 raft_consensus.cc:604] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 1 LEADER]: Received request to transfer leadership to TS 39d7bd8c4f38482b8548151b5f717bcf
I20250831 07:57:24.257699 30698 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "24dbfaab9a7a42ffa64a24fa6f384740"
dest_uuid: "357750ee400f42f8a09a88487e8bd247"
mode: GRACEFUL
new_leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:37486
I20250831 07:57:24.258155 30698 raft_consensus.cc:604] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Received request to transfer leadership to TS 39d7bd8c4f38482b8548151b5f717bcf
I20250831 07:57:24.269353 30847 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66"
dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
mode: GRACEFUL
new_leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:49182
I20250831 07:57:24.270102 30847 raft_consensus.cc:604] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 3 LEADER]: Received request to transfer leadership to TS 39d7bd8c4f38482b8548151b5f717bcf
I20250831 07:57:24.525456 31022 raft_consensus.cc:991] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e: : Instructing follower 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:24.525904 31022 raft_consensus.cc:1079] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 3 LEADER]: Signalling peer 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:24.527300 30774 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:43448
I20250831 07:57:24.527742 30774 raft_consensus.cc:491] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:24.528045 30774 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:24.536652 31024 raft_consensus.cc:991] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e: : Instructing follower 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:24.537071 31024 raft_consensus.cc:1079] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 3 LEADER]: Signalling peer 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:24.538370 30773 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "47a48a1d49224a53a0949ef880502472"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:43448
I20250831 07:57:24.538759 30773 raft_consensus.cc:491] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:24.539000 30773 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:24.631520 30774 raft_consensus.cc:513] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.634917 30774 leader_election.cc:290] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 4 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:24.636390 30698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:24.636734 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:24.637181 31025 raft_consensus.cc:991] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247: : Instructing follower 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:24.637540 30698 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:24.637669 30847 raft_consensus.cc:3053] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:57:24.637916 31008 raft_consensus.cc:1079] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 1 LEADER]: Signalling peer 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:24.638062 30847 raft_consensus.cc:738] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:24.638793 30847 consensus_queue.cc:260] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.639945 30847 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:24.640324 30774 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "fffdf93425de4d52b8d33fd7cb7ecb9a"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:43442
I20250831 07:57:24.640866 30774 raft_consensus.cc:491] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:24.641208 30774 raft_consensus.cc:3058] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:24.722568 31025 raft_consensus.cc:991] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247: : Instructing follower 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:24.723012 31008 raft_consensus.cc:1079] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Signalling peer 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:24.724319 30772 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:43442
I20250831 07:57:24.724797 30772 raft_consensus.cc:491] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:24.725100 30772 raft_consensus.cc:3058] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:24.732204 30847 raft_consensus.cc:2466] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 4.
I20250831 07:57:24.732275 30773 raft_consensus.cc:513] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.734002 30698 raft_consensus.cc:2466] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 4.
I20250831 07:57:24.735261 30773 leader_election.cc:290] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 4 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:24.736218 30740 leader_election.cc:304] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d7bd8c4f38482b8548151b5f717bcf, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:24.736940 30698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a48a1d49224a53a0949ef880502472" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:24.737617 30698 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:24.737811 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a48a1d49224a53a0949ef880502472" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:24.738413 30847 raft_consensus.cc:3053] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:57:24.738759 30847 raft_consensus.cc:738] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:24.739097 31026 raft_consensus.cc:2802] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:57:24.739621 31026 raft_consensus.cc:695] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 LEADER]: Becoming Leader. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:24.739578 30847 consensus_queue.cc:260] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.740978 30847 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:24.740677 31026 consensus_queue.cc:237] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.748052 30595 catalog_manager.cc:5582] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf reported cstate change: term changed from 3 to 4, leader changed from 4598d92c93d34ad5b95e906a309af48e (127.29.217.3) to 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2). New cstate: current_term: 4 leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:24.753000 31025 raft_consensus.cc:991] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247: : Instructing follower 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:24.753495 31008 raft_consensus.cc:1079] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Signalling peer 39d7bd8c4f38482b8548151b5f717bcf to start an election
I20250831 07:57:24.754870 30773 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "24dbfaab9a7a42ffa64a24fa6f384740"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
from {username='slave'} at 127.0.0.1:43442
I20250831 07:57:24.755447 30773 raft_consensus.cc:491] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:24.755872 30773 raft_consensus.cc:3058] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:24.832932 30698 raft_consensus.cc:2466] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 4.
I20250831 07:57:24.832978 30774 raft_consensus.cc:513] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.833305 30772 raft_consensus.cc:513] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.832932 30847 raft_consensus.cc:2466] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 4.
I20250831 07:57:24.835167 30774 leader_election.cc:290] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 2 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:24.835201 30772 leader_election.cc:290] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 3 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:24.836805 30738 leader_election.cc:304] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 39d7bd8c4f38482b8548151b5f717bcf; no voters:
I20250831 07:57:24.837766 30698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fffdf93425de4d52b8d33fd7cb7ecb9a" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:24.838198 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:24.838694 31026 raft_consensus.cc:2802] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:57:24.838284 30698 raft_consensus.cc:3053] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:24.839154 30698 raft_consensus.cc:738] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Running, Role: LEADER
I20250831 07:57:24.839231 31026 raft_consensus.cc:695] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 LEADER]: Becoming Leader. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:24.839841 30698 consensus_queue.cc:260] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.840099 31026 consensus_queue.cc:237] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.840674 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fffdf93425de4d52b8d33fd7cb7ecb9a" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:24.841042 30698 raft_consensus.cc:3058] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:24.841159 30847 raft_consensus.cc:3058] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:24.842063 30848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:24.842559 30848 raft_consensus.cc:3058] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:24.838717 30699 raft_consensus.cc:3053] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:57:24.843955 30699 raft_consensus.cc:738] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Running, Role: LEADER
I20250831 07:57:24.844648 30699 consensus_queue.cc:260] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.845818 30699 raft_consensus.cc:3058] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:24.855625 30595 catalog_manager.cc:5582] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf reported cstate change: term changed from 3 to 4, leader changed from 4598d92c93d34ad5b95e906a309af48e (127.29.217.3) to 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2). New cstate: current_term: 4 leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:24.933004 30848 raft_consensus.cc:2466] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 3.
I20250831 07:57:24.933004 30698 raft_consensus.cc:2466] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 2.
I20250831 07:57:24.933044 30773 raft_consensus.cc:513] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.934259 30847 raft_consensus.cc:2466] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 2.
I20250831 07:57:24.933290 30699 raft_consensus.cc:2466] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 3.
I20250831 07:57:24.934697 30738 leader_election.cc:304] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 39d7bd8c4f38482b8548151b5f717bcf; no voters:
I20250831 07:57:24.935851 30773 leader_election.cc:290] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 3 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:24.935938 30738 leader_election.cc:304] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 39d7bd8c4f38482b8548151b5f717bcf; no voters:
I20250831 07:57:24.936409 30699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24dbfaab9a7a42ffa64a24fa6f384740" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:24.936820 30925 raft_consensus.cc:2802] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:24.936962 30699 raft_consensus.cc:3053] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:57:24.937254 30699 raft_consensus.cc:738] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Running, Role: LEADER
I20250831 07:57:24.937747 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24dbfaab9a7a42ffa64a24fa6f384740" candidate_uuid: "39d7bd8c4f38482b8548151b5f717bcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:24.937915 31026 raft_consensus.cc:2802] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:57:24.938482 31026 raft_consensus.cc:695] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 3 LEADER]: Becoming Leader. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:24.938884 30925 raft_consensus.cc:695] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 2 LEADER]: Becoming Leader. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:24.938705 30699 consensus_queue.cc:260] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.938329 30847 raft_consensus.cc:3058] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:24.939347 31026 consensus_queue.cc:237] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.939926 30699 raft_consensus.cc:3058] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:24.939756 30925 consensus_queue.cc:237] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:24.948845 30595 catalog_manager.cc:5582] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf reported cstate change: term changed from 1 to 2, leader changed from 357750ee400f42f8a09a88487e8bd247 (127.29.217.1) to 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2). New cstate: current_term: 2 leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:24.958963 30595 catalog_manager.cc:5582] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf reported cstate change: term changed from 2 to 3, leader changed from 357750ee400f42f8a09a88487e8bd247 (127.29.217.1) to 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2). New cstate: current_term: 3 leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:25.036808 30699 raft_consensus.cc:2466] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 3.
I20250831 07:57:25.036808 30847 raft_consensus.cc:2466] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d7bd8c4f38482b8548151b5f717bcf in term 3.
I20250831 07:57:25.038244 30738 leader_election.cc:304] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 39d7bd8c4f38482b8548151b5f717bcf; no voters:
I20250831 07:57:25.039065 31026 raft_consensus.cc:2802] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:57:25.039580 31026 raft_consensus.cc:695] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 LEADER]: Becoming Leader. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:25.040565 31026 consensus_queue.cc:237] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:25.048789 30595 catalog_manager.cc:5582] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf reported cstate change: term changed from 2 to 3, leader changed from 357750ee400f42f8a09a88487e8bd247 (127.29.217.1) to 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2). New cstate: current_term: 3 leader_uuid: "39d7bd8c4f38482b8548151b5f717bcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:25.237932 30847 raft_consensus.cc:1273] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 4 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:25.238996 30925 consensus_queue.cc:1035] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:57:25.248416 30699 raft_consensus.cc:1273] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 4 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:25.250324 30925 consensus_queue.cc:1035] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:57:25.258687 30847 raft_consensus.cc:1273] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 4 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:25.259680 30925 consensus_queue.cc:1035] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:57:25.267796 30699 raft_consensus.cc:1273] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 4 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:25.269248 31033 consensus_queue.cc:1035] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:57:25.396153 30699 raft_consensus.cc:1273] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:25.397132 31033 consensus_queue.cc:1035] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:25.404829 30847 raft_consensus.cc:1273] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:25.406558 30925 consensus_queue.cc:1035] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:25.518005 30847 raft_consensus.cc:1273] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:25.519125 30925 consensus_queue.cc:1035] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:25.527263 30699 raft_consensus.cc:1273] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:25.529559 30925 consensus_queue.cc:1035] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:25.532601 30698 raft_consensus.cc:1273] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:25.533973 30925 consensus_queue.cc:1035] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:25.543056 30847 raft_consensus.cc:1273] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Refusing update from remote peer 39d7bd8c4f38482b8548151b5f717bcf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:25.545359 31033 consensus_queue.cc:1035] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:27.301924 30564 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20250831 07:57:27.302845 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:27.309625 30773 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
mode: GRACEFUL
new_leader_uuid: "4598d92c93d34ad5b95e906a309af48e"
from {username='slave'} at 127.0.0.1:43462
I20250831 07:57:27.310256 30773 raft_consensus.cc:604] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 3 LEADER]: Received request to transfer leadership to TS 4598d92c93d34ad5b95e906a309af48e
I20250831 07:57:27.312978 30773 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "27b19174fafc4d6bb0f98dc93d13dda4"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
mode: GRACEFUL
new_leader_uuid: "4598d92c93d34ad5b95e906a309af48e"
from {username='slave'} at 127.0.0.1:43462
I20250831 07:57:27.313418 30773 raft_consensus.cc:604] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 LEADER]: Received request to transfer leadership to TS 4598d92c93d34ad5b95e906a309af48e
I20250831 07:57:27.315810 30773 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "47a48a1d49224a53a0949ef880502472"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
mode: GRACEFUL
new_leader_uuid: "357750ee400f42f8a09a88487e8bd247"
from {username='slave'} at 127.0.0.1:43462
I20250831 07:57:27.316303 30773 raft_consensus.cc:604] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 LEADER]: Received request to transfer leadership to TS 357750ee400f42f8a09a88487e8bd247
I20250831 07:57:27.318715 30773 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
mode: GRACEFUL
new_leader_uuid: "357750ee400f42f8a09a88487e8bd247"
from {username='slave'} at 127.0.0.1:43462
I20250831 07:57:27.319293 30773 raft_consensus.cc:604] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 LEADER]: Received request to transfer leadership to TS 357750ee400f42f8a09a88487e8bd247
I20250831 07:57:27.321483 30773 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8524e0bbc5a04120a1862c3f2b492c3f"
dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
mode: GRACEFUL
new_leader_uuid: "357750ee400f42f8a09a88487e8bd247"
from {username='slave'} at 127.0.0.1:43462
I20250831 07:57:27.321926 30773 raft_consensus.cc:604] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 1 LEADER]: Received request to transfer leadership to TS 357750ee400f42f8a09a88487e8bd247
I20250831 07:57:27.322860 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250831 07:57:27.323264 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:27.638952 31050 raft_consensus.cc:991] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf: : Instructing follower 357750ee400f42f8a09a88487e8bd247 to start an election
I20250831 07:57:27.639418 31053 raft_consensus.cc:1079] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 LEADER]: Signalling peer 357750ee400f42f8a09a88487e8bd247 to start an election
I20250831 07:57:27.640825 30698 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "47a48a1d49224a53a0949ef880502472"
dest_uuid: "357750ee400f42f8a09a88487e8bd247"
from {username='slave'} at 127.0.0.1:37468
I20250831 07:57:27.641196 30698 raft_consensus.cc:491] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:27.641443 30698 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:57:27.646181 30698 raft_consensus.cc:513] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.647970 30698 leader_election.cc:290] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 5 election: Requested vote from peers 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:27.648882 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a48a1d49224a53a0949ef880502472" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:27.649140 30847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47a48a1d49224a53a0949ef880502472" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:27.649490 30773 raft_consensus.cc:3053] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:57:27.649592 30847 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:57:27.649748 30773 raft_consensus.cc:738] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:27.650529 30773 consensus_queue.cc:260] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.651681 30773 raft_consensus.cc:3058] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:57:27.654338 31050 raft_consensus.cc:991] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf: : Instructing follower 4598d92c93d34ad5b95e906a309af48e to start an election
I20250831 07:57:27.654796 31053 raft_consensus.cc:1079] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 LEADER]: Signalling peer 4598d92c93d34ad5b95e906a309af48e to start an election
I20250831 07:57:27.655522 30847 raft_consensus.cc:2466] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 5.
I20250831 07:57:27.656386 30848 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "27b19174fafc4d6bb0f98dc93d13dda4"
dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
from {username='slave'} at 127.0.0.1:49170
I20250831 07:57:27.656795 30665 leader_election.cc:304] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:27.656910 30848 raft_consensus.cc:491] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:27.657249 30773 raft_consensus.cc:2466] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 5.
I20250831 07:57:27.657352 30848 raft_consensus.cc:3058] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:27.658545 31054 raft_consensus.cc:2802] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:57:27.659648 31054 raft_consensus.cc:695] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 5 LEADER]: Becoming Leader. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Running, Role: LEADER
I20250831 07:57:27.660559 31054 consensus_queue.cc:237] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.661976 30848 raft_consensus.cc:513] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.664645 30848 leader_election.cc:290] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 3 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:27.665993 30698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27b19174fafc4d6bb0f98dc93d13dda4" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:27.666029 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27b19174fafc4d6bb0f98dc93d13dda4" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:27.666508 30698 raft_consensus.cc:3058] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:27.666612 30773 raft_consensus.cc:3053] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:57:27.666910 30773 raft_consensus.cc:738] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:27.667477 30773 consensus_queue.cc:260] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.668370 30773 raft_consensus.cc:3058] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:27.668221 30595 catalog_manager.cc:5582] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 reported cstate change: term changed from 4 to 5, leader changed from 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2) to 357750ee400f42f8a09a88487e8bd247 (127.29.217.1). New cstate: current_term: 5 leader_uuid: "357750ee400f42f8a09a88487e8bd247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:27.672891 30698 raft_consensus.cc:2466] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 3.
I20250831 07:57:27.672961 30773 raft_consensus.cc:2466] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 3.
I20250831 07:57:27.674182 30812 leader_election.cc:304] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:27.675010 31024 raft_consensus.cc:2802] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:57:27.676378 31024 raft_consensus.cc:695] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 3 LEADER]: Becoming Leader. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:27.677495 31024 consensus_queue.cc:237] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.686836 31050 raft_consensus.cc:991] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf: : Instructing follower 357750ee400f42f8a09a88487e8bd247 to start an election
I20250831 07:57:27.687219 31053 raft_consensus.cc:1079] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 LEADER]: Signalling peer 357750ee400f42f8a09a88487e8bd247 to start an election
I20250831 07:57:27.687036 30594 catalog_manager.cc:5582] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e reported cstate change: term changed from 2 to 3, leader changed from 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2) to 4598d92c93d34ad5b95e906a309af48e (127.29.217.3). New cstate: current_term: 3 leader_uuid: "4598d92c93d34ad5b95e906a309af48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:27.688640 30698 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66"
dest_uuid: "357750ee400f42f8a09a88487e8bd247"
from {username='slave'} at 127.0.0.1:37468
I20250831 07:57:27.689069 30698 raft_consensus.cc:491] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:27.689312 30698 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:57:27.694461 30698 raft_consensus.cc:513] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.695998 30698 leader_election.cc:290] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 5 election: Requested vote from peers 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:27.696601 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:27.696879 30848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fbb315d02b6426b8531915f088cce66" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:27.697085 30773 raft_consensus.cc:3053] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:57:27.697335 30848 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:57:27.697338 30773 raft_consensus.cc:738] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:27.698068 30773 consensus_queue.cc:260] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.698876 30773 raft_consensus.cc:3058] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:57:27.701328 30848 raft_consensus.cc:2466] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 5.
I20250831 07:57:27.702131 30665 leader_election.cc:304] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:27.702584 30773 raft_consensus.cc:2466] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 5.
I20250831 07:57:27.702704 31054 raft_consensus.cc:2802] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:57:27.703099 31054 raft_consensus.cc:695] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 5 LEADER]: Becoming Leader. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Running, Role: LEADER
I20250831 07:57:27.703897 31054 consensus_queue.cc:237] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.709846 30594 catalog_manager.cc:5582] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 reported cstate change: term changed from 4 to 5, leader changed from 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2) to 357750ee400f42f8a09a88487e8bd247 (127.29.217.1). New cstate: current_term: 5 leader_uuid: "357750ee400f42f8a09a88487e8bd247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:27.713143 31050 raft_consensus.cc:991] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf: : Instructing follower 4598d92c93d34ad5b95e906a309af48e to start an election
I20250831 07:57:27.713634 31050 raft_consensus.cc:1079] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 3 LEADER]: Signalling peer 4598d92c93d34ad5b95e906a309af48e to start an election
I20250831 07:57:27.715031 30848 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b"
dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
from {username='slave'} at 127.0.0.1:49170
I20250831 07:57:27.715504 30848 raft_consensus.cc:491] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:27.715750 30848 raft_consensus.cc:3058] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:27.720212 30848 raft_consensus.cc:513] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.721487 30848 leader_election.cc:290] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 4 election: Requested vote from peers 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467), 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785)
I20250831 07:57:27.722354 30698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "357750ee400f42f8a09a88487e8bd247"
I20250831 07:57:27.722357 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c786bfc839247baafd87289b48ca14b" candidate_uuid: "4598d92c93d34ad5b95e906a309af48e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:27.722916 30698 raft_consensus.cc:3058] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:27.722985 30773 raft_consensus.cc:3053] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:57:27.723273 30773 raft_consensus.cc:738] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:27.723862 30773 consensus_queue.cc:260] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.724939 30773 raft_consensus.cc:3058] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:27.727305 30698 raft_consensus.cc:2466] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 4.
I20250831 07:57:27.728109 30812 leader_election.cc:304] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:27.728720 31024 raft_consensus.cc:2802] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:57:27.729060 31024 raft_consensus.cc:695] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 4 LEADER]: Becoming Leader. State: Replica: 4598d92c93d34ad5b95e906a309af48e, State: Running, Role: LEADER
I20250831 07:57:27.729728 30773 raft_consensus.cc:2466] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4598d92c93d34ad5b95e906a309af48e in term 4.
I20250831 07:57:27.729863 31024 consensus_queue.cc:237] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.736174 30594 catalog_manager.cc:5582] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e reported cstate change: term changed from 3 to 4, leader changed from 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2) to 4598d92c93d34ad5b95e906a309af48e (127.29.217.3). New cstate: current_term: 4 leader_uuid: "4598d92c93d34ad5b95e906a309af48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:27.798060 31053 raft_consensus.cc:991] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf: : Instructing follower 357750ee400f42f8a09a88487e8bd247 to start an election
I20250831 07:57:27.798519 31050 raft_consensus.cc:1079] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 1 LEADER]: Signalling peer 357750ee400f42f8a09a88487e8bd247 to start an election
I20250831 07:57:27.799896 30698 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8524e0bbc5a04120a1862c3f2b492c3f"
dest_uuid: "357750ee400f42f8a09a88487e8bd247"
from {username='slave'} at 127.0.0.1:37468
I20250831 07:57:27.800395 30698 raft_consensus.cc:491] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:27.800674 30698 raft_consensus.cc:3058] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:27.804560 30698 raft_consensus.cc:513] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.805945 30698 leader_election.cc:290] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 2 election: Requested vote from peers 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785), 4598d92c93d34ad5b95e906a309af48e (127.29.217.3:43897)
I20250831 07:57:27.806923 30773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8524e0bbc5a04120a1862c3f2b492c3f" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39d7bd8c4f38482b8548151b5f717bcf"
I20250831 07:57:27.807178 30848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8524e0bbc5a04120a1862c3f2b492c3f" candidate_uuid: "357750ee400f42f8a09a88487e8bd247" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4598d92c93d34ad5b95e906a309af48e"
I20250831 07:57:27.807466 30773 raft_consensus.cc:3053] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:27.807633 30848 raft_consensus.cc:3058] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:27.807749 30773 raft_consensus.cc:738] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39d7bd8c4f38482b8548151b5f717bcf, State: Running, Role: LEADER
I20250831 07:57:27.808446 30773 consensus_queue.cc:260] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.809372 30773 raft_consensus.cc:3058] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:27.811743 30848 raft_consensus.cc:2466] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 2.
I20250831 07:57:27.812593 30665 leader_election.cc:304] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357750ee400f42f8a09a88487e8bd247, 4598d92c93d34ad5b95e906a309af48e; no voters:
I20250831 07:57:27.813171 31054 raft_consensus.cc:2802] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:27.813603 31054 raft_consensus.cc:695] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Becoming Leader. State: Replica: 357750ee400f42f8a09a88487e8bd247, State: Running, Role: LEADER
I20250831 07:57:27.814745 30773 raft_consensus.cc:2466] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357750ee400f42f8a09a88487e8bd247 in term 2.
I20250831 07:57:27.814400 31054 consensus_queue.cc:237] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } }
I20250831 07:57:27.821759 30594 catalog_manager.cc:5582] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 reported cstate change: term changed from 1 to 2, leader changed from 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2) to 357750ee400f42f8a09a88487e8bd247 (127.29.217.1). New cstate: current_term: 2 leader_uuid: "357750ee400f42f8a09a88487e8bd247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:28.075520 30698 raft_consensus.cc:1273] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Refusing update from remote peer 4598d92c93d34ad5b95e906a309af48e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:28.076553 31024 consensus_queue.cc:1035] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:28.088794 30773 raft_consensus.cc:1273] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Refusing update from remote peer 4598d92c93d34ad5b95e906a309af48e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:28.090044 31024 consensus_queue.cc:1035] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:28.149677 30848 raft_consensus.cc:1273] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 5 FOLLOWER]: Refusing update from remote peer 357750ee400f42f8a09a88487e8bd247: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 07:57:28.150821 31054 consensus_queue.cc:1035] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:57:28.160753 30773 raft_consensus.cc:1273] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 5 FOLLOWER]: Refusing update from remote peer 357750ee400f42f8a09a88487e8bd247: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 07:57:28.162084 31073 consensus_queue.cc:1035] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:57:28.180029 30848 raft_consensus.cc:1273] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 5 FOLLOWER]: Refusing update from remote peer 357750ee400f42f8a09a88487e8bd247: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 07:57:28.181108 31073 consensus_queue.cc:1035] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:57:28.189769 30773 raft_consensus.cc:1273] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 5 FOLLOWER]: Refusing update from remote peer 357750ee400f42f8a09a88487e8bd247: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 07:57:28.191174 31054 consensus_queue.cc:1035] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:57:28.215147 30848 raft_consensus.cc:1273] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Refusing update from remote peer 357750ee400f42f8a09a88487e8bd247: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:28.216213 31054 consensus_queue.cc:1035] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4598d92c93d34ad5b95e906a309af48e" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:28.224584 30773 raft_consensus.cc:1273] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Refusing update from remote peer 357750ee400f42f8a09a88487e8bd247: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:28.226133 31054 consensus_queue.cc:1035] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:28.281253 30773 raft_consensus.cc:1273] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 4 FOLLOWER]: Refusing update from remote peer 4598d92c93d34ad5b95e906a309af48e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:28.282310 31024 consensus_queue.cc:1035] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d7bd8c4f38482b8548151b5f717bcf" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:57:28.292362 30699 raft_consensus.cc:1273] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 4 FOLLOWER]: Refusing update from remote peer 4598d92c93d34ad5b95e906a309af48e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:28.293576 31059 consensus_queue.cc:1035] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "357750ee400f42f8a09a88487e8bd247" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:57:28.324429 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:28.329628 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:28.330056 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:29.330992 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:29.335667 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:29.336110 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:30.337083 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:30.341838 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:30.342255 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:31.343281 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:31.348096 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:31.348541 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:32.349606 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:32.354611 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:32.355015 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:33.356091 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:33.360780 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:33.361177 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:34.362152 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:34.367349 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:34.367797 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:35.368774 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:35.374092 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:35.374444 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:36.375411 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:36.379989 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:36.380374 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:37.406080 30564 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20250831 07:57:37.406678 30564 tablet_server.cc:178] TabletServer@127.29.217.1:0 shutting down...
W20250831 07:57:37.429922 30812 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250831 07:57:37.431617 30564 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250831 07:57:37.432556 30564 tablet_replica.cc:331] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247: stopping tablet replica
I20250831 07:57:37.433553 30564 raft_consensus.cc:2241] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 5 LEADER]: Raft consensus shutting down.
W20250831 07:57:37.433823 30812 consensus_peers.cc:489] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e -> Peer 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467): Couldn't send request to peer 357750ee400f42f8a09a88487e8bd247. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:57:37.434696 30564 raft_consensus.cc:2270] T 7fbb315d02b6426b8531915f088cce66 P 357750ee400f42f8a09a88487e8bd247 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.437108 30564 tablet_replica.cc:331] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247: stopping tablet replica
I20250831 07:57:37.437630 30564 raft_consensus.cc:2241] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.438037 30564 raft_consensus.cc:2270] T 24dbfaab9a7a42ffa64a24fa6f384740 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.439606 30564 tablet_replica.cc:331] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247: stopping tablet replica
I20250831 07:57:37.440125 30564 raft_consensus.cc:2241] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.440563 30564 raft_consensus.cc:2270] T 68688b732e6a47858fa4d8257adf011f P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.442092 30564 tablet_replica.cc:331] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247: stopping tablet replica
I20250831 07:57:37.442566 30564 raft_consensus.cc:2241] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 5 LEADER]: Raft consensus shutting down.
I20250831 07:57:37.443276 30564 raft_consensus.cc:2270] T 47a48a1d49224a53a0949ef880502472 P 357750ee400f42f8a09a88487e8bd247 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.444929 30564 tablet_replica.cc:331] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247: stopping tablet replica
I20250831 07:57:37.445375 30564 raft_consensus.cc:2241] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 2 LEADER]: Raft consensus shutting down.
I20250831 07:57:37.446102 30564 raft_consensus.cc:2270] T 8524e0bbc5a04120a1862c3f2b492c3f P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.448366 30564 tablet_replica.cc:331] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247: stopping tablet replica
I20250831 07:57:37.449023 30564 raft_consensus.cc:2241] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.449573 30564 raft_consensus.cc:2270] T 27b19174fafc4d6bb0f98dc93d13dda4 P 357750ee400f42f8a09a88487e8bd247 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.451380 30564 tablet_replica.cc:331] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247: stopping tablet replica
I20250831 07:57:37.451853 30564 raft_consensus.cc:2241] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.452232 30564 raft_consensus.cc:2270] T fffdf93425de4d52b8d33fd7cb7ecb9a P 357750ee400f42f8a09a88487e8bd247 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.453788 30564 tablet_replica.cc:331] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247: stopping tablet replica
I20250831 07:57:37.454197 30564 raft_consensus.cc:2241] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.454545 30564 raft_consensus.cc:2270] T d0b161ad222348bea9cf99d0d2dc923d P 357750ee400f42f8a09a88487e8bd247 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.456169 30564 tablet_replica.cc:331] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247: stopping tablet replica
I20250831 07:57:37.456566 30564 raft_consensus.cc:2241] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.456961 30564 raft_consensus.cc:2270] T 1c786bfc839247baafd87289b48ca14b P 357750ee400f42f8a09a88487e8bd247 [term 4 FOLLOWER]: Raft consensus is shut down!
W20250831 07:57:37.571990 30738 consensus_peers.cc:489] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf -> Peer 357750ee400f42f8a09a88487e8bd247 (127.29.217.1:33467): Couldn't send request to peer 357750ee400f42f8a09a88487e8bd247. Status: Network error: Client connection negotiation failed: client connection to 127.29.217.1:33467: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250831 07:57:37.591295 30564 tablet_server.cc:195] TabletServer@127.29.217.1:0 shutdown complete.
I20250831 07:57:37.611227 30564 tablet_server.cc:178] TabletServer@127.29.217.2:0 shutting down...
I20250831 07:57:37.637477 30564 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250831 07:57:37.638156 30564 tablet_replica.cc:331] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf: stopping tablet replica
I20250831 07:57:37.639137 30564 raft_consensus.cc:2241] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 LEADER]: Raft consensus shutting down.
I20250831 07:57:37.640054 30564 raft_consensus.cc:2270] T 24dbfaab9a7a42ffa64a24fa6f384740 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.642014 30564 tablet_replica.cc:331] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf: stopping tablet replica
I20250831 07:57:37.642493 30564 raft_consensus.cc:2241] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.642910 30564 raft_consensus.cc:2270] T 1c786bfc839247baafd87289b48ca14b P 39d7bd8c4f38482b8548151b5f717bcf [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.644455 30564 tablet_replica.cc:331] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf: stopping tablet replica
I20250831 07:57:37.644914 30564 raft_consensus.cc:2241] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [term 1 LEADER]: Raft consensus shutting down.
I20250831 07:57:37.645658 30564 raft_consensus.cc:2270] T d0b161ad222348bea9cf99d0d2dc923d P 39d7bd8c4f38482b8548151b5f717bcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.647270 30564 tablet_replica.cc:331] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf: stopping tablet replica
I20250831 07:57:37.647734 30564 raft_consensus.cc:2241] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 5 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.648211 30564 raft_consensus.cc:2270] T 7fbb315d02b6426b8531915f088cce66 P 39d7bd8c4f38482b8548151b5f717bcf [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.649792 30564 tablet_replica.cc:331] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf: stopping tablet replica
I20250831 07:57:37.650224 30564 raft_consensus.cc:2241] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.650591 30564 raft_consensus.cc:2270] T 68688b732e6a47858fa4d8257adf011f P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.652403 30564 tablet_replica.cc:331] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf: stopping tablet replica
I20250831 07:57:37.652856 30564 raft_consensus.cc:2241] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 5 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.653209 30564 raft_consensus.cc:2270] T 47a48a1d49224a53a0949ef880502472 P 39d7bd8c4f38482b8548151b5f717bcf [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.654711 30564 tablet_replica.cc:331] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf: stopping tablet replica
I20250831 07:57:37.655153 30564 raft_consensus.cc:2241] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.655510 30564 raft_consensus.cc:2270] T 27b19174fafc4d6bb0f98dc93d13dda4 P 39d7bd8c4f38482b8548151b5f717bcf [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.656975 30564 tablet_replica.cc:331] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf: stopping tablet replica
I20250831 07:57:37.657408 30564 raft_consensus.cc:2241] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 2 LEADER]: Raft consensus shutting down.
I20250831 07:57:37.658092 30564 raft_consensus.cc:2270] T fffdf93425de4d52b8d33fd7cb7ecb9a P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.659417 30564 tablet_replica.cc:331] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf: stopping tablet replica
I20250831 07:57:37.659842 30564 raft_consensus.cc:2241] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.660208 30564 raft_consensus.cc:2270] T 8524e0bbc5a04120a1862c3f2b492c3f P 39d7bd8c4f38482b8548151b5f717bcf [term 2 FOLLOWER]: Raft consensus is shut down!
W20250831 07:57:37.680321 30811 consensus_peers.cc:489] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e -> Peer 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785): Couldn't send request to peer 39d7bd8c4f38482b8548151b5f717bcf. Status: Network error: Client connection negotiation failed: client connection to 127.29.217.2:41785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250831 07:57:37.687507 30564 tablet_server.cc:195] TabletServer@127.29.217.2:0 shutdown complete.
I20250831 07:57:37.706964 30564 tablet_server.cc:178] TabletServer@127.29.217.3:0 shutting down...
W20250831 07:57:37.714927 30811 consensus_peers.cc:489] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e -> Peer 39d7bd8c4f38482b8548151b5f717bcf (127.29.217.2:41785): Couldn't send request to peer 39d7bd8c4f38482b8548151b5f717bcf. Status: Network error: Client connection negotiation failed: client connection to 127.29.217.2:41785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250831 07:57:37.731751 30564 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250831 07:57:37.732450 30564 tablet_replica.cc:331] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e: stopping tablet replica
I20250831 07:57:37.733043 30564 raft_consensus.cc:2241] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.733485 30564 raft_consensus.cc:2270] T 24dbfaab9a7a42ffa64a24fa6f384740 P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.735155 30564 tablet_replica.cc:331] T d0b161ad222348bea9cf99d0d2dc923d P 4598d92c93d34ad5b95e906a309af48e: stopping tablet replica
I20250831 07:57:37.735611 30564 raft_consensus.cc:2241] T d0b161ad222348bea9cf99d0d2dc923d P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.736002 30564 raft_consensus.cc:2270] T d0b161ad222348bea9cf99d0d2dc923d P 4598d92c93d34ad5b95e906a309af48e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.737562 30564 tablet_replica.cc:331] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e: stopping tablet replica
I20250831 07:57:37.737989 30564 raft_consensus.cc:2241] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 4 LEADER]: Raft consensus shutting down.
I20250831 07:57:37.738725 30564 raft_consensus.cc:2270] T 1c786bfc839247baafd87289b48ca14b P 4598d92c93d34ad5b95e906a309af48e [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.740320 30564 tablet_replica.cc:331] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e: stopping tablet replica
I20250831 07:57:37.740734 30564 raft_consensus.cc:2241] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 2 LEADER]: Raft consensus shutting down.
I20250831 07:57:37.741770 30564 raft_consensus.cc:2270] T 68688b732e6a47858fa4d8257adf011f P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.743294 30564 tablet_replica.cc:331] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e: stopping tablet replica
I20250831 07:57:37.743732 30564 raft_consensus.cc:2241] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.744138 30564 raft_consensus.cc:2270] T 8524e0bbc5a04120a1862c3f2b492c3f P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.745679 30564 tablet_replica.cc:331] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e: stopping tablet replica
I20250831 07:57:37.746088 30564 raft_consensus.cc:2241] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 3 LEADER]: Raft consensus shutting down.
I20250831 07:57:37.746769 30564 raft_consensus.cc:2270] T 27b19174fafc4d6bb0f98dc93d13dda4 P 4598d92c93d34ad5b95e906a309af48e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.748313 30564 tablet_replica.cc:331] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e: stopping tablet replica
I20250831 07:57:37.748720 30564 raft_consensus.cc:2241] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.749050 30564 raft_consensus.cc:2270] T fffdf93425de4d52b8d33fd7cb7ecb9a P 4598d92c93d34ad5b95e906a309af48e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.750463 30564 tablet_replica.cc:331] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e: stopping tablet replica
I20250831 07:57:37.750849 30564 raft_consensus.cc:2241] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 5 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.751197 30564 raft_consensus.cc:2270] T 7fbb315d02b6426b8531915f088cce66 P 4598d92c93d34ad5b95e906a309af48e [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.752712 30564 tablet_replica.cc:331] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e: stopping tablet replica
I20250831 07:57:37.753078 30564 raft_consensus.cc:2241] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 5 FOLLOWER]: Raft consensus shutting down.
I20250831 07:57:37.753412 30564 raft_consensus.cc:2270] T 47a48a1d49224a53a0949ef880502472 P 4598d92c93d34ad5b95e906a309af48e [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.780189 30564 tablet_server.cc:195] TabletServer@127.29.217.3:0 shutdown complete.
I20250831 07:57:37.798235 30564 master.cc:561] Master@127.29.217.62:34243 shutting down...
I20250831 07:57:37.981189 30564 raft_consensus.cc:2241] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [term 1 LEADER]: Raft consensus shutting down.
I20250831 07:57:37.981778 30564 raft_consensus.cc:2270] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:57:37.982100 30564 tablet_replica.cc:331] T 00000000000000000000000000000000 P cdc80dd77c5d4b50a69ab18e290f99a1: stopping tablet replica
I20250831 07:57:38.000658 30564 master.cc:583] Master@127.29.217.62:34243 shutdown complete.
357750ee400f42f8a09a88487e8bd247 4
39d7bd8c4f38482b8548151b5f717bcf 4
4598d92c93d34ad5b95e906a309af48e 1
357750ee400f42f8a09a88487e8bd247 3
39d7bd8c4f38482b8548151b5f717bcf 3
4598d92c93d34ad5b95e906a309af48e 3
357750ee400f42f8a09a88487e8bd247 0
39d7bd8c4f38482b8548151b5f717bcf 8
4598d92c93d34ad5b95e906a309af48e 1
357750ee400f42f8a09a88487e8bd247 3
39d7bd8c4f38482b8548151b5f717bcf 3
4598d92c93d34ad5b95e906a309af48e 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31329 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20250831 07:57:38.041625 30564 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.217.62:34445
I20250831 07:57:38.042725 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 07:57:38.048635 31095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:57:38.049374 31094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:57:38.049576 31097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:57:38.050836 30564 server_base.cc:1047] running on GCE node
I20250831 07:57:38.051795 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 07:57:38.052033 30564 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 07:57:38.052176 30564 hybrid_clock.cc:648] HybridClock initialized: now 1756627058052159 us; error 0 us; skew 500 ppm
I20250831 07:57:38.052681 30564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 07:57:38.054991 30564 webserver.cc:480] Webserver started at http://127.29.217.62:42973/ using document root <none> and password file <none>
I20250831 07:57:38.055466 30564 fs_manager.cc:362] Metadata directory not provided
I20250831 07:57:38.055651 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 07:57:38.055939 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 07:57:38.057054 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/master-0-root/instance:
uuid: "c3e825111c04425db0218a70b3653bf0"
format_stamp: "Formatted at 2025-08-31 07:57:38 on dist-test-slave-87tg"
I20250831 07:57:38.061684 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250831 07:57:38.064932 31102 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:38.065676 30564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250831 07:57:38.065953 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/master-0-root
uuid: "c3e825111c04425db0218a70b3653bf0"
format_stamp: "Formatted at 2025-08-31 07:57:38 on dist-test-slave-87tg"
I20250831 07:57:38.066210 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 07:57:38.083351 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 07:57:38.084619 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 07:57:38.121737 30564 rpc_server.cc:307] RPC server started. Bound to: 127.29.217.62:34445
I20250831 07:57:38.121835 31153 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.217.62:34445 every 8 connection(s)
I20250831 07:57:38.125360 31154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.134518 31154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0: Bootstrap starting.
I20250831 07:57:38.138995 31154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:38.143163 31154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0: No bootstrap required, opened a new log
I20250831 07:57:38.145197 31154 raft_consensus.cc:357] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e825111c04425db0218a70b3653bf0" member_type: VOTER }
I20250831 07:57:38.145651 31154 raft_consensus.cc:383] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:38.145905 31154 raft_consensus.cc:738] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e825111c04425db0218a70b3653bf0, State: Initialized, Role: FOLLOWER
I20250831 07:57:38.146463 31154 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e825111c04425db0218a70b3653bf0" member_type: VOTER }
I20250831 07:57:38.146912 31154 raft_consensus.cc:397] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250831 07:57:38.147128 31154 raft_consensus.cc:491] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250831 07:57:38.147392 31154 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:38.151232 31154 raft_consensus.cc:513] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e825111c04425db0218a70b3653bf0" member_type: VOTER }
I20250831 07:57:38.151701 31154 leader_election.cc:304] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3e825111c04425db0218a70b3653bf0; no voters:
I20250831 07:57:38.152798 31154 leader_election.cc:290] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250831 07:57:38.153100 31157 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:38.154373 31157 raft_consensus.cc:695] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [term 1 LEADER]: Becoming Leader. State: Replica: c3e825111c04425db0218a70b3653bf0, State: Running, Role: LEADER
I20250831 07:57:38.155042 31157 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e825111c04425db0218a70b3653bf0" member_type: VOTER }
I20250831 07:57:38.155673 31154 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 07:57:38.159339 31158 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3e825111c04425db0218a70b3653bf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e825111c04425db0218a70b3653bf0" member_type: VOTER } }
I20250831 07:57:38.159627 31159 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3e825111c04425db0218a70b3653bf0. Latest consensus state: current_term: 1 leader_uuid: "c3e825111c04425db0218a70b3653bf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e825111c04425db0218a70b3653bf0" member_type: VOTER } }
I20250831 07:57:38.160027 31158 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [sys.catalog]: This master's current role is: LEADER
I20250831 07:57:38.160331 31159 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [sys.catalog]: This master's current role is: LEADER
I20250831 07:57:38.163362 31163 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250831 07:57:38.171247 31163 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250831 07:57:38.172201 30564 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250831 07:57:38.179548 31163 catalog_manager.cc:1349] Generated new cluster ID: 190549282f9a41faaba343e3ab05cb29
I20250831 07:57:38.179880 31163 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250831 07:57:38.196758 31163 catalog_manager.cc:1372] Generated new certificate authority record
I20250831 07:57:38.197961 31163 catalog_manager.cc:1506] Loading token signing keys...
I20250831 07:57:38.217842 31163 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0: Generated new TSK 0
I20250831 07:57:38.218442 31163 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250831 07:57:38.239537 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 07:57:38.245862 31175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:57:38.247103 31176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:57:38.248660 31178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:57:38.249238 30564 server_base.cc:1047] running on GCE node
I20250831 07:57:38.250223 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 07:57:38.250432 30564 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 07:57:38.250609 30564 hybrid_clock.cc:648] HybridClock initialized: now 1756627058250590 us; error 0 us; skew 500 ppm
I20250831 07:57:38.251248 30564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 07:57:38.253813 30564 webserver.cc:480] Webserver started at http://127.29.217.1:34187/ using document root <none> and password file <none>
I20250831 07:57:38.254336 30564 fs_manager.cc:362] Metadata directory not provided
I20250831 07:57:38.254514 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 07:57:38.254771 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 07:57:38.255791 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-0-root/instance:
uuid: "2961283b97774862959c98dba71d8d78"
format_stamp: "Formatted at 2025-08-31 07:57:38 on dist-test-slave-87tg"
I20250831 07:57:38.260282 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250831 07:57:38.263302 31183 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:38.264045 30564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250831 07:57:38.264343 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-0-root
uuid: "2961283b97774862959c98dba71d8d78"
format_stamp: "Formatted at 2025-08-31 07:57:38 on dist-test-slave-87tg"
I20250831 07:57:38.264626 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 07:57:38.277622 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 07:57:38.278821 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 07:57:38.280217 30564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 07:57:38.282322 30564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 07:57:38.282506 30564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:38.282753 30564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 07:57:38.282923 30564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:38.322091 30564 rpc_server.cc:307] RPC server started. Bound to: 127.29.217.1:43687
I20250831 07:57:38.322180 31245 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.217.1:43687 every 8 connection(s)
I20250831 07:57:38.326534 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 07:57:38.333575 31250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:57:38.334528 31251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:57:38.338379 31253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:57:38.338553 30564 server_base.cc:1047] running on GCE node
I20250831 07:57:38.339465 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 07:57:38.339640 30564 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 07:57:38.339766 30564 hybrid_clock.cc:648] HybridClock initialized: now 1756627058339754 us; error 0 us; skew 500 ppm
I20250831 07:57:38.340251 30564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 07:57:38.342304 30564 webserver.cc:480] Webserver started at http://127.29.217.2:36681/ using document root <none> and password file <none>
I20250831 07:57:38.342829 30564 fs_manager.cc:362] Metadata directory not provided
I20250831 07:57:38.343051 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 07:57:38.343359 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 07:57:38.344470 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-1-root/instance:
uuid: "e175aaa0dcf3486984c542aaf3563df1"
format_stamp: "Formatted at 2025-08-31 07:57:38 on dist-test-slave-87tg"
I20250831 07:57:38.348572 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250831 07:57:38.351545 31258 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:38.352263 30564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250831 07:57:38.352550 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-1-root
uuid: "e175aaa0dcf3486984c542aaf3563df1"
format_stamp: "Formatted at 2025-08-31 07:57:38 on dist-test-slave-87tg"
I20250831 07:57:38.352828 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 07:57:38.363375 31246 heartbeater.cc:344] Connected to a master server at 127.29.217.62:34445
I20250831 07:57:38.363785 31246 heartbeater.cc:461] Registering TS with master...
I20250831 07:57:38.364508 31246 heartbeater.cc:507] Master 127.29.217.62:34445 requested a full tablet report, sending...
I20250831 07:57:38.366572 31119 ts_manager.cc:194] Registered new tserver with Master: 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687)
I20250831 07:57:38.368870 31119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34612
I20250831 07:57:38.376282 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 07:57:38.377467 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 07:57:38.378856 30564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 07:57:38.381099 30564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 07:57:38.381273 30564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:38.381510 30564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 07:57:38.381649 30564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:38.420161 30564 rpc_server.cc:307] RPC server started. Bound to: 127.29.217.2:44001
I20250831 07:57:38.420327 31320 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.217.2:44001 every 8 connection(s)
I20250831 07:57:38.424742 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 07:57:38.431151 31324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:57:38.433868 31325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:57:38.437409 30564 server_base.cc:1047] running on GCE node
I20250831 07:57:38.437597 31321 heartbeater.cc:344] Connected to a master server at 127.29.217.62:34445
W20250831 07:57:38.437436 31327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:57:38.438133 31321 heartbeater.cc:461] Registering TS with master...
I20250831 07:57:38.438656 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 07:57:38.438889 30564 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 07:57:38.438943 31321 heartbeater.cc:507] Master 127.29.217.62:34445 requested a full tablet report, sending...
I20250831 07:57:38.439051 30564 hybrid_clock.cc:648] HybridClock initialized: now 1756627058439034 us; error 0 us; skew 500 ppm
I20250831 07:57:38.439659 30564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 07:57:38.441031 31119 ts_manager.cc:194] Registered new tserver with Master: e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001)
I20250831 07:57:38.442225 30564 webserver.cc:480] Webserver started at http://127.29.217.3:38291/ using document root <none> and password file <none>
I20250831 07:57:38.442495 31119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34614
I20250831 07:57:38.442744 30564 fs_manager.cc:362] Metadata directory not provided
I20250831 07:57:38.442965 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 07:57:38.443208 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 07:57:38.444494 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-2-root/instance:
uuid: "e59304d21a1e44febcc404d724fc1577"
format_stamp: "Formatted at 2025-08-31 07:57:38 on dist-test-slave-87tg"
I20250831 07:57:38.449101 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250831 07:57:38.452493 31332 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:38.453239 30564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20250831 07:57:38.453532 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-2-root
uuid: "e59304d21a1e44febcc404d724fc1577"
format_stamp: "Formatted at 2025-08-31 07:57:38 on dist-test-slave-87tg"
I20250831 07:57:38.453809 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756627026548763-30564-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 07:57:38.490927 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 07:57:38.492194 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 07:57:38.493635 30564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 07:57:38.495872 30564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 07:57:38.496088 30564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:38.496311 30564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 07:57:38.496475 30564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 07:57:38.538400 30564 rpc_server.cc:307] RPC server started. Bound to: 127.29.217.3:36251
I20250831 07:57:38.538488 31394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.217.3:36251 every 8 connection(s)
I20250831 07:57:38.553212 31395 heartbeater.cc:344] Connected to a master server at 127.29.217.62:34445
I20250831 07:57:38.553588 31395 heartbeater.cc:461] Registering TS with master...
I20250831 07:57:38.554315 31395 heartbeater.cc:507] Master 127.29.217.62:34445 requested a full tablet report, sending...
I20250831 07:57:38.556264 31119 ts_manager.cc:194] Registered new tserver with Master: e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:38.556388 30564 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014815927s
I20250831 07:57:38.557636 31119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34616
I20250831 07:57:38.560168 30564 test_util.cc:276] Using random seed: -1680601975
I20250831 07:57:38.586103 31119 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34618:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250831 07:57:38.589268 31119 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250831 07:57:38.663090 31284 tablet_service.cc:1468] Processing CreateTablet for tablet c6698eb6af2f4c888796b8e9f1003baf (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250831 07:57:38.664292 31286 tablet_service.cc:1468] Processing CreateTablet for tablet d9002d7d65a14e49ad6c842c00ced7f8 (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250831 07:57:38.665158 31284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6698eb6af2f4c888796b8e9f1003baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.667316 31285 tablet_service.cc:1468] Processing CreateTablet for tablet e2415c634a154f269bf1f47ba35a97c3 (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250831 07:57:38.668898 31285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2415c634a154f269bf1f47ba35a97c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.673630 31283 tablet_service.cc:1468] Processing CreateTablet for tablet 7800f3d155894012ab8b8a3067479d5f (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250831 07:57:38.675249 31283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7800f3d155894012ab8b8a3067479d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.676985 31282 tablet_service.cc:1468] Processing CreateTablet for tablet 3db868229f2e47d0af6034b101d57e7a (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250831 07:57:38.678408 31282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db868229f2e47d0af6034b101d57e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.681087 31281 tablet_service.cc:1468] Processing CreateTablet for tablet acc1b872a4834f68a4494a11d4487df5 (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250831 07:57:38.682394 31281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acc1b872a4834f68a4494a11d4487df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.685091 31280 tablet_service.cc:1468] Processing CreateTablet for tablet e9ad3c047ea14d648689485283a199aa (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250831 07:57:38.686368 31280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9ad3c047ea14d648689485283a199aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.688599 31286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9002d7d65a14e49ad6c842c00ced7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.697000 31211 tablet_service.cc:1468] Processing CreateTablet for tablet d9002d7d65a14e49ad6c842c00ced7f8 (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250831 07:57:38.698482 31211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9002d7d65a14e49ad6c842c00ced7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.698347 31279 tablet_service.cc:1468] Processing CreateTablet for tablet 0d2c02218be24f76ac0c207781177c0c (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250831 07:57:38.700482 31279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2c02218be24f76ac0c207781177c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.701130 31210 tablet_service.cc:1468] Processing CreateTablet for tablet e2415c634a154f269bf1f47ba35a97c3 (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250831 07:57:38.702466 31210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2415c634a154f269bf1f47ba35a97c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.705183 31206 tablet_service.cc:1468] Processing CreateTablet for tablet acc1b872a4834f68a4494a11d4487df5 (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250831 07:57:38.705176 31205 tablet_service.cc:1468] Processing CreateTablet for tablet e9ad3c047ea14d648689485283a199aa (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250831 07:57:38.706511 31206 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acc1b872a4834f68a4494a11d4487df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.708022 31204 tablet_service.cc:1468] Processing CreateTablet for tablet 0d2c02218be24f76ac0c207781177c0c (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250831 07:57:38.709331 31204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2c02218be24f76ac0c207781177c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.709017 31207 tablet_service.cc:1468] Processing CreateTablet for tablet 3db868229f2e47d0af6034b101d57e7a (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250831 07:57:38.710243 31207 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db868229f2e47d0af6034b101d57e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.712955 31208 tablet_service.cc:1468] Processing CreateTablet for tablet 7800f3d155894012ab8b8a3067479d5f (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250831 07:57:38.714360 31208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7800f3d155894012ab8b8a3067479d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.716154 31205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9ad3c047ea14d648689485283a199aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.717046 31209 tablet_service.cc:1468] Processing CreateTablet for tablet c6698eb6af2f4c888796b8e9f1003baf (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250831 07:57:38.718308 31209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6698eb6af2f4c888796b8e9f1003baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.732111 31278 tablet_service.cc:1468] Processing CreateTablet for tablet b46e28f6af7247e182075273233a24bc (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250831 07:57:38.733433 31203 tablet_service.cc:1468] Processing CreateTablet for tablet b46e28f6af7247e182075273233a24bc (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250831 07:57:38.734287 31278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b46e28f6af7247e182075273233a24bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.734928 31203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b46e28f6af7247e182075273233a24bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.737118 31202 tablet_service.cc:1468] Processing CreateTablet for tablet 32f97fdb66fd4c24bcdef839b4c3432a (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250831 07:57:38.737682 31277 tablet_service.cc:1468] Processing CreateTablet for tablet 32f97fdb66fd4c24bcdef839b4c3432a (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250831 07:57:38.738492 31202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32f97fdb66fd4c24bcdef839b4c3432a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.739591 31277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32f97fdb66fd4c24bcdef839b4c3432a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.785804 31360 tablet_service.cc:1468] Processing CreateTablet for tablet d9002d7d65a14e49ad6c842c00ced7f8 (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250831 07:57:38.787267 31360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9002d7d65a14e49ad6c842c00ced7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.789829 31354 tablet_service.cc:1468] Processing CreateTablet for tablet e9ad3c047ea14d648689485283a199aa (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250831 07:57:38.790052 31359 tablet_service.cc:1468] Processing CreateTablet for tablet e2415c634a154f269bf1f47ba35a97c3 (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250831 07:57:38.791242 31354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9ad3c047ea14d648689485283a199aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.791911 31359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2415c634a154f269bf1f47ba35a97c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.793128 31358 tablet_service.cc:1468] Processing CreateTablet for tablet c6698eb6af2f4c888796b8e9f1003baf (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250831 07:57:38.793891 31353 tablet_service.cc:1468] Processing CreateTablet for tablet 0d2c02218be24f76ac0c207781177c0c (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250831 07:57:38.797046 31356 tablet_service.cc:1468] Processing CreateTablet for tablet 3db868229f2e47d0af6034b101d57e7a (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250831 07:57:38.798352 31356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db868229f2e47d0af6034b101d57e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.800978 31355 tablet_service.cc:1468] Processing CreateTablet for tablet acc1b872a4834f68a4494a11d4487df5 (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250831 07:57:38.802281 31355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acc1b872a4834f68a4494a11d4487df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.803771 31353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2c02218be24f76ac0c207781177c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.795513 31357 tablet_service.cc:1468] Processing CreateTablet for tablet 7800f3d155894012ab8b8a3067479d5f (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250831 07:57:38.805966 31357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7800f3d155894012ab8b8a3067479d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.808259 31358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6698eb6af2f4c888796b8e9f1003baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.858381 31353 tablet_service.cc:1468] Processing CreateTablet for tablet b46e28f6af7247e182075273233a24bc (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250831 07:57:38.859931 31353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b46e28f6af7247e182075273233a24bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.861472 31352 tablet_service.cc:1468] Processing CreateTablet for tablet 32f97fdb66fd4c24bcdef839b4c3432a (DEFAULT_TABLE table=test-workload [id=323562ed25154fdc83795470aa6ab64c]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250831 07:57:38.863147 31352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32f97fdb66fd4c24bcdef839b4c3432a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:57:38.881229 31410 tablet_bootstrap.cc:492] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1: Bootstrap starting.
I20250831 07:57:38.887641 31410 tablet_bootstrap.cc:654] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:38.897369 31411 tablet_bootstrap.cc:492] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78: Bootstrap starting.
I20250831 07:57:38.903247 31412 tablet_bootstrap.cc:492] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577: Bootstrap starting.
I20250831 07:57:38.906606 31411 tablet_bootstrap.cc:654] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:38.918689 31412 tablet_bootstrap.cc:654] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:38.925482 31410 tablet_bootstrap.cc:492] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1: No bootstrap required, opened a new log
I20250831 07:57:38.926059 31410 ts_tablet_manager.cc:1397] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1: Time spent bootstrapping tablet: real 0.048s user 0.013s sys 0.008s
I20250831 07:57:38.927014 31411 tablet_bootstrap.cc:492] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78: No bootstrap required, opened a new log
I20250831 07:57:38.927600 31411 ts_tablet_manager.cc:1397] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.000s
I20250831 07:57:38.929414 31412 tablet_bootstrap.cc:492] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577: No bootstrap required, opened a new log
I20250831 07:57:38.929028 31410 raft_consensus.cc:357] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:38.929613 31411 raft_consensus.cc:357] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:38.929888 31412 ts_tablet_manager.cc:1397] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.000s
I20250831 07:57:38.929827 31410 raft_consensus.cc:383] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:38.930245 31411 raft_consensus.cc:383] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:38.930402 31410 raft_consensus.cc:738] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Initialized, Role: FOLLOWER
I20250831 07:57:38.930605 31411 raft_consensus.cc:738] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Initialized, Role: FOLLOWER
I20250831 07:57:38.931252 31411 consensus_queue.cc:260] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:38.931231 31410 consensus_queue.cc:260] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:38.932300 31412 raft_consensus.cc:357] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:38.933110 31412 raft_consensus.cc:383] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:38.933416 31412 raft_consensus.cc:738] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Initialized, Role: FOLLOWER
I20250831 07:57:38.934280 31412 consensus_queue.cc:260] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:38.939525 31412 ts_tablet_manager.cc:1428] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20250831 07:57:38.941725 31411 ts_tablet_manager.cc:1428] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20250831 07:57:38.941903 31410 ts_tablet_manager.cc:1428] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1: Time spent starting tablet: real 0.015s user 0.005s sys 0.008s
I20250831 07:57:38.942873 31246 heartbeater.cc:499] Master 127.29.217.62:34445 was elected leader, sending a full tablet report...
I20250831 07:57:38.943279 31410 tablet_bootstrap.cc:492] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1: Bootstrap starting.
I20250831 07:57:38.943348 31395 heartbeater.cc:499] Master 127.29.217.62:34445 was elected leader, sending a full tablet report...
I20250831 07:57:38.944325 31412 tablet_bootstrap.cc:492] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577: Bootstrap starting.
I20250831 07:57:38.947687 31321 heartbeater.cc:499] Master 127.29.217.62:34445 was elected leader, sending a full tablet report...
I20250831 07:57:38.950057 31410 tablet_bootstrap.cc:654] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:38.951236 31412 tablet_bootstrap.cc:654] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:38.957026 31411 tablet_bootstrap.cc:492] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78: Bootstrap starting.
I20250831 07:57:38.964406 31411 tablet_bootstrap.cc:654] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.071641 31412 tablet_bootstrap.cc:492] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577: No bootstrap required, opened a new log
I20250831 07:57:39.072322 31412 ts_tablet_manager.cc:1397] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577: Time spent bootstrapping tablet: real 0.128s user 0.027s sys 0.091s
I20250831 07:57:39.075727 31412 raft_consensus.cc:357] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.076366 31412 raft_consensus.cc:383] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.076601 31412 raft_consensus.cc:738] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.077216 31412 consensus_queue.cc:260] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.079303 31410 tablet_bootstrap.cc:492] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1: No bootstrap required, opened a new log
I20250831 07:57:39.079523 31412 ts_tablet_manager.cc:1428] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20250831 07:57:39.079931 31410 ts_tablet_manager.cc:1397] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1: Time spent bootstrapping tablet: real 0.137s user 0.050s sys 0.081s
I20250831 07:57:39.080447 31412 tablet_bootstrap.cc:492] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577: Bootstrap starting.
I20250831 07:57:39.080710 31411 tablet_bootstrap.cc:492] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78: No bootstrap required, opened a new log
I20250831 07:57:39.081184 31411 ts_tablet_manager.cc:1397] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78: Time spent bootstrapping tablet: real 0.124s user 0.044s sys 0.069s
I20250831 07:57:39.082805 31410 raft_consensus.cc:357] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.083616 31410 raft_consensus.cc:383] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.083719 31411 raft_consensus.cc:357] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.084051 31410 raft_consensus.cc:738] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.084451 31411 raft_consensus.cc:383] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.084777 31411 raft_consensus.cc:738] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.085134 31410 consensus_queue.cc:260] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.085489 31411 consensus_queue.cc:260] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.087779 31411 ts_tablet_manager.cc:1428] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250831 07:57:39.088690 31411 tablet_bootstrap.cc:492] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78: Bootstrap starting.
I20250831 07:57:39.089274 31412 tablet_bootstrap.cc:654] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.092877 31410 ts_tablet_manager.cc:1428] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250831 07:57:39.095093 31411 tablet_bootstrap.cc:654] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.097230 31410 tablet_bootstrap.cc:492] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1: Bootstrap starting.
I20250831 07:57:39.106094 31410 tablet_bootstrap.cc:654] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.109005 31412 tablet_bootstrap.cc:492] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577: No bootstrap required, opened a new log
I20250831 07:57:39.109508 31412 ts_tablet_manager.cc:1397] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.013s
I20250831 07:57:39.111773 31411 tablet_bootstrap.cc:492] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78: No bootstrap required, opened a new log
I20250831 07:57:39.112275 31411 ts_tablet_manager.cc:1397] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.019s
I20250831 07:57:39.112790 31412 raft_consensus.cc:357] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.113730 31412 raft_consensus.cc:383] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.114111 31412 raft_consensus.cc:738] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.114877 31411 raft_consensus.cc:357] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.115554 31411 raft_consensus.cc:383] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.115803 31411 raft_consensus.cc:738] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.115161 31412 consensus_queue.cc:260] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.116644 31411 consensus_queue.cc:260] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.117918 31412 ts_tablet_manager.cc:1428] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250831 07:57:39.118824 31411 ts_tablet_manager.cc:1428] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20250831 07:57:39.118894 31412 tablet_bootstrap.cc:492] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577: Bootstrap starting.
I20250831 07:57:39.119756 31411 tablet_bootstrap.cc:492] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78: Bootstrap starting.
I20250831 07:57:39.125574 31412 tablet_bootstrap.cc:654] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.126094 31411 tablet_bootstrap.cc:654] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.126987 31410 tablet_bootstrap.cc:492] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1: No bootstrap required, opened a new log
I20250831 07:57:39.127452 31410 ts_tablet_manager.cc:1397] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.006s
I20250831 07:57:39.130847 31410 raft_consensus.cc:357] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.131803 31410 raft_consensus.cc:383] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.132202 31410 raft_consensus.cc:738] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.133177 31410 consensus_queue.cc:260] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.135500 31416 raft_consensus.cc:491] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.136490 31416 raft_consensus.cc:513] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.137562 31410 ts_tablet_manager.cc:1428] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20250831 07:57:39.139093 31416 leader_election.cc:290] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.139590 31412 tablet_bootstrap.cc:492] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577: No bootstrap required, opened a new log
I20250831 07:57:39.140406 31411 tablet_bootstrap.cc:492] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78: No bootstrap required, opened a new log
I20250831 07:57:39.140893 31411 ts_tablet_manager.cc:1397] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.000s
I20250831 07:57:39.141885 31412 ts_tablet_manager.cc:1397] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.000s
I20250831 07:57:39.143543 31411 raft_consensus.cc:357] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.144260 31411 raft_consensus.cc:383] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.144555 31411 raft_consensus.cc:738] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.144520 31412 raft_consensus.cc:357] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.145305 31412 raft_consensus.cc:383] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.145637 31412 raft_consensus.cc:738] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.146103 31410 tablet_bootstrap.cc:492] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1: Bootstrap starting.
I20250831 07:57:39.146309 31412 consensus_queue.cc:260] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.148944 31412 ts_tablet_manager.cc:1428] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250831 07:57:39.149943 31412 tablet_bootstrap.cc:492] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577: Bootstrap starting.
I20250831 07:57:39.152256 31410 tablet_bootstrap.cc:654] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.145325 31411 consensus_queue.cc:260] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.154430 31411 ts_tablet_manager.cc:1428] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20250831 07:57:39.155527 31411 tablet_bootstrap.cc:492] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78: Bootstrap starting.
I20250831 07:57:39.165823 31411 tablet_bootstrap.cc:654] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78: Neither blocks nor log segments found. Creating new log.
W20250831 07:57:39.166999 31167 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 07:57:39.172300 31412 tablet_bootstrap.cc:654] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.174753 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
W20250831 07:57:39.176582 31185 leader_election.cc:343] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.180631 31411 tablet_bootstrap.cc:492] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78: No bootstrap required, opened a new log
I20250831 07:57:39.181190 31411 ts_tablet_manager.cc:1397] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.000s
I20250831 07:57:39.181910 31416 raft_consensus.cc:491] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.182374 31416 raft_consensus.cc:513] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.184373 31416 leader_election.cc:290] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.184309 31411 raft_consensus.cc:357] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.185184 31411 raft_consensus.cc:383] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.185557 31411 raft_consensus.cc:738] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.186038 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
I20250831 07:57:39.186322 31411 consensus_queue.cc:260] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
W20250831 07:57:39.187472 31185 leader_election.cc:343] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.188886 31411 ts_tablet_manager.cc:1428] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250831 07:57:39.189944 31411 tablet_bootstrap.cc:492] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78: Bootstrap starting.
I20250831 07:57:39.196102 31411 tablet_bootstrap.cc:654] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.208361 31416 raft_consensus.cc:491] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.208564 31411 tablet_bootstrap.cc:492] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78: No bootstrap required, opened a new log
I20250831 07:57:39.209103 31411 ts_tablet_manager.cc:1397] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.003s
I20250831 07:57:39.208905 31416 raft_consensus.cc:513] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.209040 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.209861 31410 tablet_bootstrap.cc:492] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1: No bootstrap required, opened a new log
I20250831 07:57:39.209820 31370 raft_consensus.cc:2466] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 0.
I20250831 07:57:39.210352 31410 ts_tablet_manager.cc:1397] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1: Time spent bootstrapping tablet: real 0.064s user 0.020s sys 0.040s
I20250831 07:57:39.211112 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.211542 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
I20250831 07:57:39.212013 31411 raft_consensus.cc:357] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.212692 31411 raft_consensus.cc:383] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250831 07:57:39.212812 31185 leader_election.cc:343] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.213001 31411 raft_consensus.cc:738] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.213254 31184 leader_election.cc:304] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577; no voters: e175aaa0dcf3486984c542aaf3563df1
I20250831 07:57:39.213687 31416 leader_election.cc:290] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.214239 31416 raft_consensus.cc:2802] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:39.213809 31411 consensus_queue.cc:260] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.214588 31416 raft_consensus.cc:491] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:39.214535 31417 raft_consensus.cc:491] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.215139 31416 raft_consensus.cc:3058] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.215303 31417 raft_consensus.cc:513] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.216440 31411 ts_tablet_manager.cc:1428] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250831 07:57:39.218502 31417 leader_election.cc:290] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.213044 31410 raft_consensus.cc:357] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.219209 31410 raft_consensus.cc:383] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.219621 31410 raft_consensus.cc:738] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.220715 31410 consensus_queue.cc:260] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.223636 31410 ts_tablet_manager.cc:1428] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20250831 07:57:39.224601 31410 tablet_bootstrap.cc:492] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1: Bootstrap starting.
I20250831 07:57:39.224669 31411 tablet_bootstrap.cc:492] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78: Bootstrap starting.
I20250831 07:57:39.231161 31410 tablet_bootstrap.cc:654] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.232867 31411 tablet_bootstrap.cc:654] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.243146 31412 tablet_bootstrap.cc:492] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577: No bootstrap required, opened a new log
I20250831 07:57:39.255960 31412 ts_tablet_manager.cc:1397] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577: Time spent bootstrapping tablet: real 0.094s user 0.022s sys 0.019s
I20250831 07:57:39.262616 31416 raft_consensus.cc:513] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.267256 31417 raft_consensus.cc:491] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.267928 31417 raft_consensus.cc:513] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.270504 31417 leader_election.cc:290] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
W20250831 07:57:39.282480 31184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:36251, user_credentials={real_user=slave}} blocked reactor thread for 69507.3us
W20250831 07:57:39.283809 31184 leader_election.cc:343] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.284579 31184 leader_election.cc:304] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2961283b97774862959c98dba71d8d78; no voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577
I20250831 07:57:39.288334 31412 raft_consensus.cc:357] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.289400 31412 raft_consensus.cc:383] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.289769 31412 raft_consensus.cc:738] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.293108 31439 raft_consensus.cc:491] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.308717 31433 raft_consensus.cc:2747] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:57:39.295470 31439 raft_consensus.cc:513] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.315471 31439 leader_election.cc:290] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.325259 31416 leader_election.cc:290] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.329264 31412 consensus_queue.cc:260] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.333884 31410 tablet_bootstrap.cc:492] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1: No bootstrap required, opened a new log
I20250831 07:57:39.334373 31410 ts_tablet_manager.cc:1397] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1: Time spent bootstrapping tablet: real 0.110s user 0.044s sys 0.059s
I20250831 07:57:39.334568 31411 tablet_bootstrap.cc:492] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78: No bootstrap required, opened a new log
I20250831 07:57:39.335000 31411 ts_tablet_manager.cc:1397] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78: Time spent bootstrapping tablet: real 0.111s user 0.040s sys 0.068s
I20250831 07:57:39.334213 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.337231 31410 raft_consensus.cc:357] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.337972 31410 raft_consensus.cc:383] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.337641 31411 raft_consensus.cc:357] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.338271 31410 raft_consensus.cc:738] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.338371 31411 raft_consensus.cc:383] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.338696 31411 raft_consensus.cc:738] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.339131 31410 consensus_queue.cc:260] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.339350 31411 consensus_queue.cc:260] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.340417 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:39.340977 31296 raft_consensus.cc:3058] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.341959 31411 ts_tablet_manager.cc:1428] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250831 07:57:39.342970 31411 tablet_bootstrap.cc:492] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78: Bootstrap starting.
I20250831 07:57:39.346179 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2415c634a154f269bf1f47ba35a97c3" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.346848 31369 raft_consensus.cc:2466] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 0.
I20250831 07:57:39.348299 31296 raft_consensus.cc:2466] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 1.
I20250831 07:57:39.348377 31221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2415c634a154f269bf1f47ba35a97c3" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78" is_pre_election: true
I20250831 07:57:39.349119 31221 raft_consensus.cc:2466] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 0.
I20250831 07:57:39.349377 31411 tablet_bootstrap.cc:654] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.350210 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:39.350339 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78" is_pre_election: true
I20250831 07:57:39.350553 31185 leader_election.cc:304] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1; no voters:
I20250831 07:57:39.350836 31369 raft_consensus.cc:3058] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.350222 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2415c634a154f269bf1f47ba35a97c3" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
I20250831 07:57:39.351861 31296 raft_consensus.cc:2466] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 0.
I20250831 07:57:39.352893 31438 raft_consensus.cc:2802] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:39.353379 31438 raft_consensus.cc:695] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Becoming Leader. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:39.354238 31438 consensus_queue.cc:237] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.357715 31410 ts_tablet_manager.cc:1428] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1: Time spent starting tablet: real 0.023s user 0.005s sys 0.004s
I20250831 07:57:39.358935 31369 raft_consensus.cc:2466] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 1.
I20250831 07:57:39.365195 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2415c634a154f269bf1f47ba35a97c3" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.365738 31370 raft_consensus.cc:2466] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 0.
I20250831 07:57:39.359592 31416 raft_consensus.cc:491] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.359694 31439 raft_consensus.cc:491] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.366490 31416 raft_consensus.cc:513] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.367192 31439 raft_consensus.cc:513] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.368492 31416 leader_election.cc:290] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.368958 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.369216 31439 leader_election.cc:290] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
W20250831 07:57:39.360347 31184 leader_election.cc:343] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.362571 31260 leader_election.cc:304] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1; no voters:
I20250831 07:57:39.368448 31410 tablet_bootstrap.cc:492] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1: Bootstrap starting.
W20250831 07:57:39.371064 31260 leader_election.cc:343] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.371996 31417 raft_consensus.cc:2802] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:39.359310 31185 leader_election.cc:304] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1; no voters:
I20250831 07:57:39.372359 31417 raft_consensus.cc:491] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:39.372749 31417 raft_consensus.cc:3058] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.374143 31418 raft_consensus.cc:491] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.374600 31418 raft_consensus.cc:513] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.377173 31418 leader_election.cc:290] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001)
I20250831 07:57:39.376888 31295 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b46e28f6af7247e182075273233a24bc" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
W20250831 07:57:39.378000 31259 leader_election.cc:343] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.378436 31259 leader_election.cc:304] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1; no voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577
I20250831 07:57:39.378108 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9002d7d65a14e49ad6c842c00ced7f8" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
W20250831 07:57:39.378798 31185 leader_election.cc:343] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.380707 31439 raft_consensus.cc:2802] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:39.381686 31439 raft_consensus.cc:491] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:39.382032 31439 raft_consensus.cc:3058] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:57:39.384799 31185 leader_election.cc:343] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.386708 31412 ts_tablet_manager.cc:1428] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577: Time spent starting tablet: real 0.130s user 0.007s sys 0.001s
I20250831 07:57:39.389003 31410 tablet_bootstrap.cc:654] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.370249 31184 leader_election.cc:304] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2961283b97774862959c98dba71d8d78; no voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577
I20250831 07:57:39.392417 31416 raft_consensus.cc:2747] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:57:39.395347 31412 tablet_bootstrap.cc:492] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577: Bootstrap starting.
I20250831 07:57:39.397099 31439 raft_consensus.cc:513] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.395362 31417 raft_consensus.cc:513] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.399998 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2415c634a154f269bf1f47ba35a97c3" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:39.401225 31296 raft_consensus.cc:2391] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2961283b97774862959c98dba71d8d78 in current term 1: Already voted for candidate e175aaa0dcf3486984c542aaf3563df1 in this term.
I20250831 07:57:39.403035 31417 leader_election.cc:290] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.403548 31417 raft_consensus.cc:2747] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:57:39.406219 31439 leader_election.cc:290] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.409013 31117 catalog_manager.cc:5582] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2961283b97774862959c98dba71d8d78 (127.29.217.1). New cstate: current_term: 1 leader_uuid: "2961283b97774862959c98dba71d8d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:39.411978 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2415c634a154f269bf1f47ba35a97c3" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:39.412732 31220 raft_consensus.cc:2391] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e175aaa0dcf3486984c542aaf3563df1 in current term 1: Already voted for candidate 2961283b97774862959c98dba71d8d78 in this term.
I20250831 07:57:39.414510 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2415c634a154f269bf1f47ba35a97c3" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:39.415076 31369 raft_consensus.cc:3058] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.419538 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9002d7d65a14e49ad6c842c00ced7f8" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.421919 31412 tablet_bootstrap.cc:654] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.422519 31411 tablet_bootstrap.cc:492] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78: No bootstrap required, opened a new log
I20250831 07:57:39.423086 31411 ts_tablet_manager.cc:1397] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78: Time spent bootstrapping tablet: real 0.080s user 0.031s sys 0.039s
W20250831 07:57:39.427291 31184 leader_election.cc:343] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.427733 31184 leader_election.cc:304] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2961283b97774862959c98dba71d8d78; no voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577
I20250831 07:57:39.429183 31368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2415c634a154f269bf1f47ba35a97c3" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:39.430754 31439 raft_consensus.cc:2747] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:57:39.425978 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b46e28f6af7247e182075273233a24bc" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.431387 31370 raft_consensus.cc:2466] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 0.
I20250831 07:57:39.432117 31184 leader_election.cc:304] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2961283b97774862959c98dba71d8d78; no voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577
I20250831 07:57:39.433884 31439 raft_consensus.cc:2747] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250831 07:57:39.435000 31184 leader_election.cc:304] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577; no voters: e175aaa0dcf3486984c542aaf3563df1
I20250831 07:57:39.435261 31369 raft_consensus.cc:2466] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 1.
I20250831 07:57:39.436543 31439 raft_consensus.cc:2802] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:39.437901 31439 raft_consensus.cc:491] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:39.438311 31439 raft_consensus.cc:3058] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.438614 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78" is_pre_election: true
I20250831 07:57:39.442731 31439 raft_consensus.cc:513] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
W20250831 07:57:39.443846 31334 leader_election.cc:343] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.444564 31259 leader_election.cc:304] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters: 2961283b97774862959c98dba71d8d78
I20250831 07:57:39.427750 31411 raft_consensus.cc:357] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.445518 31442 raft_consensus.cc:2802] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:39.444819 31439 leader_election.cc:290] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.446750 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b46e28f6af7247e182075273233a24bc" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:39.448431 31442 raft_consensus.cc:695] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 1 LEADER]: Becoming Leader. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Running, Role: LEADER
W20250831 07:57:39.449230 31185 leader_election.cc:343] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.449600 31442 consensus_queue.cc:237] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.452150 31417 raft_consensus.cc:491] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.452733 31417 raft_consensus.cc:513] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.454764 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
I20250831 07:57:39.455356 31296 raft_consensus.cc:2466] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 0.
I20250831 07:57:39.456568 31410 tablet_bootstrap.cc:492] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1: No bootstrap required, opened a new log
I20250831 07:57:39.456408 31334 leader_election.cc:304] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters: 2961283b97774862959c98dba71d8d78
I20250831 07:57:39.457450 31410 ts_tablet_manager.cc:1397] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1: Time spent bootstrapping tablet: real 0.089s user 0.027s sys 0.011s
I20250831 07:57:39.457559 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78" is_pre_election: true
I20250831 07:57:39.457921 31418 raft_consensus.cc:2802] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:39.458312 31418 raft_consensus.cc:491] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:39.458629 31418 raft_consensus.cc:3058] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:57:39.459511 31260 leader_election.cc:343] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.460314 31410 raft_consensus.cc:357] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.461022 31410 raft_consensus.cc:383] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.461303 31410 raft_consensus.cc:738] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.461661 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.461974 31410 consensus_queue.cc:260] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.463008 31417 leader_election.cc:290] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.464334 31410 ts_tablet_manager.cc:1428] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250831 07:57:39.465333 31410 tablet_bootstrap.cc:492] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1: Bootstrap starting.
I20250831 07:57:39.456844 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b46e28f6af7247e182075273233a24bc" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:39.445627 31411 raft_consensus.cc:383] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.466313 31411 raft_consensus.cc:738] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.466109 31369 raft_consensus.cc:3058] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.467209 31411 consensus_queue.cc:260] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.469643 31411 ts_tablet_manager.cc:1428] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78: Time spent starting tablet: real 0.046s user 0.007s sys 0.003s
I20250831 07:57:39.470485 31411 tablet_bootstrap.cc:492] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78: Bootstrap starting.
I20250831 07:57:39.474000 31412 tablet_bootstrap.cc:492] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577: No bootstrap required, opened a new log
I20250831 07:57:39.474767 31412 ts_tablet_manager.cc:1397] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577: Time spent bootstrapping tablet: real 0.080s user 0.017s sys 0.005s
I20250831 07:57:39.474056 31118 catalog_manager.cc:5582] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "e175aaa0dcf3486984c542aaf3563df1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:39.475997 31411 tablet_bootstrap.cc:654] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.478778 31412 raft_consensus.cc:357] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.479568 31412 raft_consensus.cc:383] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.479900 31412 raft_consensus.cc:738] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.480718 31442 raft_consensus.cc:491] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.480657 31412 consensus_queue.cc:260] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.481225 31442 raft_consensus.cc:513] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.466606 31418 raft_consensus.cc:513] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.483029 31370 raft_consensus.cc:2391] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e175aaa0dcf3486984c542aaf3563df1 in current term 1: Already voted for candidate e59304d21a1e44febcc404d724fc1577 in this term.
I20250831 07:57:39.483664 31418 leader_election.cc:290] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001)
I20250831 07:57:39.484326 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78" is_pre_election: true
I20250831 07:57:39.484988 31220 raft_consensus.cc:2466] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 0.
I20250831 07:57:39.486176 31260 leader_election.cc:304] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1; no voters:
I20250831 07:57:39.487098 31448 raft_consensus.cc:2802] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:39.487457 31448 raft_consensus.cc:491] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:39.487761 31448 raft_consensus.cc:3058] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.488796 31259 leader_election.cc:304] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1; no voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577
I20250831 07:57:39.490356 31417 raft_consensus.cc:3058] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.492399 31410 tablet_bootstrap.cc:654] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.492220 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:39.490352 31369 raft_consensus.cc:2466] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 1.
I20250831 07:57:39.498560 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:39.500205 31442 leader_election.cc:290] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.500514 31448 raft_consensus.cc:513] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.507726 31184 leader_election.cc:304] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577; no voters: e175aaa0dcf3486984c542aaf3563df1
I20250831 07:57:39.508354 31412 ts_tablet_manager.cc:1428] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577: Time spent starting tablet: real 0.033s user 0.006s sys 0.008s
I20250831 07:57:39.508992 31448 leader_election.cc:290] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.509349 31412 tablet_bootstrap.cc:492] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577: Bootstrap starting.
I20250831 07:57:39.509861 31439 raft_consensus.cc:2802] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:39.510202 31439 raft_consensus.cc:695] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Becoming Leader. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:39.510815 31439 consensus_queue.cc:237] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
W20250831 07:57:39.511503 31334 leader_election.cc:343] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.513108 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:39.513680 31220 raft_consensus.cc:3058] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.516281 31412 tablet_bootstrap.cc:654] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.517499 31410 tablet_bootstrap.cc:492] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1: No bootstrap required, opened a new log
I20250831 07:57:39.517894 31410 ts_tablet_manager.cc:1397] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.002s
I20250831 07:57:39.520279 31410 raft_consensus.cc:357] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.520977 31410 raft_consensus.cc:383] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.521265 31410 raft_consensus.cc:738] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.521667 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:39.521942 31412 tablet_bootstrap.cc:492] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577: No bootstrap required, opened a new log
I20250831 07:57:39.521998 31410 consensus_queue.cc:260] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.522564 31412 ts_tablet_manager.cc:1397] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
W20250831 07:57:39.522939 31259 leader_election.cc:343] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.524756 31410 ts_tablet_manager.cc:1428] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250831 07:57:39.525786 31410 tablet_bootstrap.cc:492] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1: Bootstrap starting.
I20250831 07:57:39.525578 31412 raft_consensus.cc:357] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.526422 31412 raft_consensus.cc:383] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.526760 31412 raft_consensus.cc:738] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.527521 31412 consensus_queue.cc:260] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.501698 31417 raft_consensus.cc:2747] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:57:39.502606 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.503475 31411 tablet_bootstrap.cc:492] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78: No bootstrap required, opened a new log
I20250831 07:57:39.531296 31411 ts_tablet_manager.cc:1397] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.021s
I20250831 07:57:39.531235 31220 raft_consensus.cc:2466] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 1.
W20250831 07:57:39.531487 31259 leader_election.cc:343] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:57:39.531394 31116 catalog_manager.cc:5582] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2961283b97774862959c98dba71d8d78 (127.29.217.1). New cstate: current_term: 1 leader_uuid: "2961283b97774862959c98dba71d8d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:39.533870 31411 raft_consensus.cc:357] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.534514 31411 raft_consensus.cc:383] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.534791 31411 raft_consensus.cc:738] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.535374 31411 consensus_queue.cc:260] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.536187 31260 leader_election.cc:304] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1; no voters: e59304d21a1e44febcc404d724fc1577
I20250831 07:57:39.530174 31412 ts_tablet_manager.cc:1428] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 07:57:39.537787 31411 ts_tablet_manager.cc:1428] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20250831 07:57:39.538440 31412 tablet_bootstrap.cc:492] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577: Bootstrap starting.
I20250831 07:57:39.538654 31411 tablet_bootstrap.cc:492] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78: Bootstrap starting.
I20250831 07:57:39.539484 31410 tablet_bootstrap.cc:654] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.546380 31411 tablet_bootstrap.cc:654] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.549024 31296 raft_consensus.cc:2466] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 1.
I20250831 07:57:39.546380 31412 tablet_bootstrap.cc:654] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.550860 31334 leader_election.cc:304] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters: 2961283b97774862959c98dba71d8d78
I20250831 07:57:39.551735 31452 raft_consensus.cc:2802] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:39.552232 31452 raft_consensus.cc:695] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Becoming Leader. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:39.552793 31411 tablet_bootstrap.cc:492] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78: No bootstrap required, opened a new log
I20250831 07:57:39.553344 31411 ts_tablet_manager.cc:1397] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.001s
I20250831 07:57:39.553054 31452 consensus_queue.cc:237] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.556929 31411 raft_consensus.cc:357] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.557929 31411 raft_consensus.cc:383] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.558262 31411 raft_consensus.cc:738] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.559798 31411 consensus_queue.cc:260] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.561853 31412 tablet_bootstrap.cc:492] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577: No bootstrap required, opened a new log
I20250831 07:57:39.562100 31411 ts_tablet_manager.cc:1428] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78: Time spent starting tablet: real 0.008s user 0.006s sys 0.003s
I20250831 07:57:39.562299 31412 ts_tablet_manager.cc:1397] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.000s
I20250831 07:57:39.563362 31417 raft_consensus.cc:2802] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:39.563949 31417 raft_consensus.cc:695] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 1 LEADER]: Becoming Leader. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Running, Role: LEADER
I20250831 07:57:39.564590 31410 tablet_bootstrap.cc:492] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1: No bootstrap required, opened a new log
I20250831 07:57:39.565011 31410 ts_tablet_manager.cc:1397] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.000s
I20250831 07:57:39.564962 31412 raft_consensus.cc:357] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.564855 31417 consensus_queue.cc:237] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.565790 31412 raft_consensus.cc:383] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.566329 31412 raft_consensus.cc:738] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.567018 31412 consensus_queue.cc:260] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.567644 31410 raft_consensus.cc:357] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.568288 31410 raft_consensus.cc:383] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.568552 31410 raft_consensus.cc:738] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.569147 31410 consensus_queue.cc:260] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.569371 31412 ts_tablet_manager.cc:1428] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250831 07:57:39.570822 31412 tablet_bootstrap.cc:492] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577: Bootstrap starting.
I20250831 07:57:39.571205 31410 ts_tablet_manager.cc:1428] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20250831 07:57:39.572019 31410 tablet_bootstrap.cc:492] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1: Bootstrap starting.
I20250831 07:57:39.576555 31412 tablet_bootstrap.cc:654] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.587328 31410 tablet_bootstrap.cc:654] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.589226 31118 catalog_manager.cc:5582] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "e175aaa0dcf3486984c542aaf3563df1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:39.589689 31117 catalog_manager.cc:5582] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59304d21a1e44febcc404d724fc1577 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "e59304d21a1e44febcc404d724fc1577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:39.591394 31452 raft_consensus.cc:491] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.591907 31452 raft_consensus.cc:513] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.592891 31410 tablet_bootstrap.cc:492] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1: No bootstrap required, opened a new log
I20250831 07:57:39.593587 31410 ts_tablet_manager.cc:1397] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20250831 07:57:39.593958 31452 leader_election.cc:290] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001)
I20250831 07:57:39.594079 31412 tablet_bootstrap.cc:492] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577: No bootstrap required, opened a new log
I20250831 07:57:39.594565 31412 ts_tablet_manager.cc:1397] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.001s
I20250831 07:57:39.596357 31410 raft_consensus.cc:357] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.597565 31410 raft_consensus.cc:383] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.597896 31410 raft_consensus.cc:738] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.598578 31410 consensus_queue.cc:260] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.598970 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78" is_pre_election: true
I20250831 07:57:39.599565 31220 raft_consensus.cc:2466] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 0.
I20250831 07:57:39.600564 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
I20250831 07:57:39.601178 31410 ts_tablet_manager.cc:1428] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250831 07:57:39.601754 31334 leader_election.cc:304] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:39.602126 31410 tablet_bootstrap.cc:492] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1: Bootstrap starting.
I20250831 07:57:39.601178 31296 raft_consensus.cc:2466] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 0.
I20250831 07:57:39.596524 31412 raft_consensus.cc:357] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.606489 31412 raft_consensus.cc:383] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.606879 31452 raft_consensus.cc:2802] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:39.606882 31412 raft_consensus.cc:738] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.607400 31452 raft_consensus.cc:491] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:39.608184 31452 raft_consensus.cc:3058] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.612726 31410 tablet_bootstrap.cc:654] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.608163 31412 consensus_queue.cc:260] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.615985 31412 ts_tablet_manager.cc:1428] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250831 07:57:39.616076 31452 raft_consensus.cc:513] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.617183 31412 tablet_bootstrap.cc:492] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577: Bootstrap starting.
I20250831 07:57:39.619763 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:39.620424 31220 raft_consensus.cc:3058] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.620507 31410 tablet_bootstrap.cc:492] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1: No bootstrap required, opened a new log
I20250831 07:57:39.620977 31410 ts_tablet_manager.cc:1397] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.000s
I20250831 07:57:39.622296 31452 leader_election.cc:290] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001)
I20250831 07:57:39.622925 31412 tablet_bootstrap.cc:654] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577: Neither blocks nor log segments found. Creating new log.
I20250831 07:57:39.623334 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:39.624101 31296 raft_consensus.cc:3058] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.624872 31410 raft_consensus.cc:357] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.625517 31410 raft_consensus.cc:383] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.625779 31410 raft_consensus.cc:738] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.626498 31410 consensus_queue.cc:260] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.630892 31296 raft_consensus.cc:2466] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 1.
I20250831 07:57:39.631134 31410 ts_tablet_manager.cc:1428] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250831 07:57:39.632300 31334 leader_election.cc:304] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:39.633132 31452 raft_consensus.cc:2802] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:39.633792 31452 raft_consensus.cc:695] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Becoming Leader. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:39.634611 31452 consensus_queue.cc:237] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.636313 31220 raft_consensus.cc:2466] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 1.
I20250831 07:57:39.640908 31412 tablet_bootstrap.cc:492] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577: No bootstrap required, opened a new log
I20250831 07:57:39.641371 31412 ts_tablet_manager.cc:1397] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.012s
I20250831 07:57:39.644322 31412 raft_consensus.cc:357] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.644960 31412 raft_consensus.cc:383] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:57:39.645223 31412 raft_consensus.cc:738] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Initialized, Role: FOLLOWER
I20250831 07:57:39.645290 31418 raft_consensus.cc:491] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.645737 31418 raft_consensus.cc:513] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.646013 31412 consensus_queue.cc:260] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.647636 31418 leader_election.cc:290] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001)
I20250831 07:57:39.648193 31412 ts_tablet_manager.cc:1428] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250831 07:57:39.648728 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
I20250831 07:57:39.650192 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78" is_pre_election: true
I20250831 07:57:39.650751 31220 raft_consensus.cc:2391] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e59304d21a1e44febcc404d724fc1577 in current term 1: Already voted for candidate e175aaa0dcf3486984c542aaf3563df1 in this term.
I20250831 07:57:39.651557 31334 leader_election.cc:304] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e59304d21a1e44febcc404d724fc1577; no voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1
I20250831 07:57:39.652295 31418 raft_consensus.cc:3058] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.653983 31117 catalog_manager.cc:5582] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59304d21a1e44febcc404d724fc1577 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "e59304d21a1e44febcc404d724fc1577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:39.658422 31418 raft_consensus.cc:2747] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 07:57:39.680140 31322 tablet.cc:2378] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250831 07:57:39.698423 31418 raft_consensus.cc:491] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.698940 31418 raft_consensus.cc:513] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.700589 31418 leader_election.cc:290] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687)
I20250831 07:57:39.701479 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9002d7d65a14e49ad6c842c00ced7f8" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
I20250831 07:57:39.701718 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9002d7d65a14e49ad6c842c00ced7f8" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78" is_pre_election: true
I20250831 07:57:39.702028 31296 raft_consensus.cc:2466] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 0.
I20250831 07:57:39.702265 31220 raft_consensus.cc:2466] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 0.
I20250831 07:57:39.702934 31334 leader_election.cc:304] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:39.703622 31418 raft_consensus.cc:2802] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:39.704007 31418 raft_consensus.cc:491] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:39.704298 31418 raft_consensus.cc:3058] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.708590 31418 raft_consensus.cc:513] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.710253 31418 leader_election.cc:290] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687)
I20250831 07:57:39.711081 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9002d7d65a14e49ad6c842c00ced7f8" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:39.711210 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9002d7d65a14e49ad6c842c00ced7f8" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:39.711524 31296 raft_consensus.cc:3058] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.711683 31220 raft_consensus.cc:3058] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.716341 31296 raft_consensus.cc:2466] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 1.
I20250831 07:57:39.716470 31220 raft_consensus.cc:2466] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 1.
I20250831 07:57:39.717298 31334 leader_election.cc:304] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:39.717945 31418 raft_consensus.cc:2802] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:39.718397 31418 raft_consensus.cc:695] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Becoming Leader. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:39.719144 31418 consensus_queue.cc:237] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.725605 31117 catalog_manager.cc:5582] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59304d21a1e44febcc404d724fc1577 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "e59304d21a1e44febcc404d724fc1577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:39.739003 31439 raft_consensus.cc:491] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.739488 31439 raft_consensus.cc:513] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.741127 31439 leader_election.cc:290] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.741961 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32f97fdb66fd4c24bcdef839b4c3432a" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
I20250831 07:57:39.742048 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32f97fdb66fd4c24bcdef839b4c3432a" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.742529 31296 raft_consensus.cc:2466] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 0.
I20250831 07:57:39.742585 31370 raft_consensus.cc:2466] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 0.
I20250831 07:57:39.743446 31184 leader_election.cc:304] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:39.744086 31439 raft_consensus.cc:2802] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:39.744431 31439 raft_consensus.cc:491] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:39.744658 31439 raft_consensus.cc:3058] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.748991 31439 raft_consensus.cc:513] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.750460 31439 leader_election.cc:290] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.751317 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32f97fdb66fd4c24bcdef839b4c3432a" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:39.751385 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32f97fdb66fd4c24bcdef839b4c3432a" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:39.751785 31296 raft_consensus.cc:3058] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.751864 31370 raft_consensus.cc:3058] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.756633 31296 raft_consensus.cc:2466] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 1.
I20250831 07:57:39.757835 31185 leader_election.cc:304] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1; no voters:
I20250831 07:57:39.758306 31370 raft_consensus.cc:2466] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 1.
I20250831 07:57:39.758628 31439 raft_consensus.cc:2802] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:39.759093 31439 raft_consensus.cc:695] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Becoming Leader. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:39.760345 31438 consensus_queue.cc:1035] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:57:39.760215 31439 consensus_queue.cc:237] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.770027 31117 catalog_manager.cc:5582] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2961283b97774862959c98dba71d8d78 (127.29.217.1). New cstate: current_term: 1 leader_uuid: "2961283b97774862959c98dba71d8d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:39.771934 31438 consensus_queue.cc:1035] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:39.809005 31438 raft_consensus.cc:491] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.809428 31438 raft_consensus.cc:513] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.811164 31438 leader_election.cc:290] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.812028 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
I20250831 07:57:39.812171 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.812862 31296 raft_consensus.cc:2391] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2961283b97774862959c98dba71d8d78 in current term 1: Already voted for candidate e59304d21a1e44febcc404d724fc1577 in this term.
I20250831 07:57:39.813740 31185 leader_election.cc:304] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2961283b97774862959c98dba71d8d78; no voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577
I20250831 07:57:39.814338 31438 raft_consensus.cc:3058] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.818852 31438 raft_consensus.cc:2747] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:57:39.871129 31438 raft_consensus.cc:491] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.871672 31438 raft_consensus.cc:513] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.872473 31417 raft_consensus.cc:491] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.872876 31417 raft_consensus.cc:513] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.874058 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
I20250831 07:57:39.874648 31296 raft_consensus.cc:2466] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 0.
I20250831 07:57:39.874588 31448 consensus_queue.cc:1035] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:39.873663 31438 leader_election.cc:290] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.875591 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b46e28f6af7247e182075273233a24bc" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.875671 31185 leader_election.cc:304] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1; no voters:
I20250831 07:57:39.876084 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577" is_pre_election: true
I20250831 07:57:39.876559 31370 raft_consensus.cc:2391] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e175aaa0dcf3486984c542aaf3563df1 in current term 1: Already voted for candidate 2961283b97774862959c98dba71d8d78 in this term.
I20250831 07:57:39.876571 31438 raft_consensus.cc:2802] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:39.876806 31369 raft_consensus.cc:2466] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 0.
I20250831 07:57:39.877149 31438 raft_consensus.cc:491] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:39.877514 31438 raft_consensus.cc:3058] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.877766 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b46e28f6af7247e182075273233a24bc" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78" is_pre_election: true
I20250831 07:57:39.875051 31417 leader_election.cc:290] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.879395 31260 leader_election.cc:304] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1; no voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577
I20250831 07:57:39.880623 31417 raft_consensus.cc:3058] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.884615 31438 raft_consensus.cc:513] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.887380 31417 raft_consensus.cc:2747] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:57:39.887603 31438 leader_election.cc:290] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:39.888567 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:39.890411 31370 raft_consensus.cc:3058] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.889883 31442 consensus_queue.cc:1035] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:57:39.897696 31296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:39.898411 31296 raft_consensus.cc:3058] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.899067 31370 raft_consensus.cc:2466] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 1.
I20250831 07:57:39.900293 31184 leader_election.cc:304] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:39.901297 31438 raft_consensus.cc:2802] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:39.901762 31438 raft_consensus.cc:695] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Becoming Leader. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:39.902580 31438 consensus_queue.cc:237] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.904574 31439 consensus_queue.cc:1035] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:39.907052 31296 raft_consensus.cc:2466] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 1.
I20250831 07:57:39.915884 31116 catalog_manager.cc:5582] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2961283b97774862959c98dba71d8d78 (127.29.217.1). New cstate: current_term: 1 leader_uuid: "2961283b97774862959c98dba71d8d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:39.922024 31439 consensus_queue.cc:1035] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:57:39.930322 31418 raft_consensus.cc:491] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:57:39.930789 31418 raft_consensus.cc:513] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.932518 31418 leader_election.cc:290] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001)
I20250831 07:57:39.933327 31221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78" is_pre_election: true
I20250831 07:57:39.933617 31295 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1" is_pre_election: true
I20250831 07:57:39.933992 31221 raft_consensus.cc:2466] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 0.
I20250831 07:57:39.934391 31295 raft_consensus.cc:2466] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 0.
I20250831 07:57:39.935586 31334 leader_election.cc:304] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:39.936465 31418 raft_consensus.cc:2802] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:57:39.937239 31418 raft_consensus.cc:491] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:57:39.937601 31418 raft_consensus.cc:3058] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.942682 31418 raft_consensus.cc:513] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.944159 31418 leader_election.cc:290] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001)
I20250831 07:57:39.944916 31221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:39.945178 31295 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:39.945359 31221 raft_consensus.cc:3058] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.945679 31295 raft_consensus.cc:3058] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:57:39.950150 31295 raft_consensus.cc:2466] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 1.
I20250831 07:57:39.950264 31221 raft_consensus.cc:2466] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 1.
I20250831 07:57:39.951122 31334 leader_election.cc:304] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:39.951862 31418 raft_consensus.cc:2802] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:57:39.952232 31418 raft_consensus.cc:695] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Becoming Leader. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:39.952849 31418 consensus_queue.cc:237] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:39.959422 31118 catalog_manager.cc:5582] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59304d21a1e44febcc404d724fc1577 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "e59304d21a1e44febcc404d724fc1577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:39.984231 31448 consensus_queue.cc:1035] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:39.993475 31476 consensus_queue.cc:1035] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:40.065891 31418 consensus_queue.cc:1035] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:40.077720 31418 consensus_queue.cc:1035] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:40.097759 31370 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d9002d7d65a14e49ad6c842c00ced7f8"
dest_uuid: "e59304d21a1e44febcc404d724fc1577"
mode: GRACEFUL
new_leader_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:38624
I20250831 07:57:40.098443 31370 raft_consensus.cc:604] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Received request to transfer leadership to TS 2961283b97774862959c98dba71d8d78
I20250831 07:57:40.112316 31295 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e2415c634a154f269bf1f47ba35a97c3"
dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
mode: GRACEFUL
new_leader_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:39510
I20250831 07:57:40.112818 31295 raft_consensus.cc:604] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 1 LEADER]: Received request to transfer leadership to TS 2961283b97774862959c98dba71d8d78
I20250831 07:57:40.119771 31370 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f"
dest_uuid: "e59304d21a1e44febcc404d724fc1577"
mode: GRACEFUL
new_leader_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:38624
I20250831 07:57:40.120291 31370 raft_consensus.cc:604] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Received request to transfer leadership to TS 2961283b97774862959c98dba71d8d78
I20250831 07:57:40.125147 31295 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a"
dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
mode: GRACEFUL
new_leader_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:39510
I20250831 07:57:40.125653 31295 raft_consensus.cc:604] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 1 LEADER]: Received request to transfer leadership to TS 2961283b97774862959c98dba71d8d78
I20250831 07:57:40.130138 31370 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5"
dest_uuid: "e59304d21a1e44febcc404d724fc1577"
mode: GRACEFUL
new_leader_uuid: "e175aaa0dcf3486984c542aaf3563df1"
from {username='slave'} at 127.0.0.1:38624
I20250831 07:57:40.130604 31370 raft_consensus.cc:604] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Received request to transfer leadership to TS e175aaa0dcf3486984c542aaf3563df1
I20250831 07:57:40.134891 31370 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa"
dest_uuid: "e59304d21a1e44febcc404d724fc1577"
mode: GRACEFUL
new_leader_uuid: "e175aaa0dcf3486984c542aaf3563df1"
from {username='slave'} at 127.0.0.1:38624
I20250831 07:57:40.135341 31370 raft_consensus.cc:604] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Received request to transfer leadership to TS e175aaa0dcf3486984c542aaf3563df1
I20250831 07:57:40.148092 31220 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c"
dest_uuid: "2961283b97774862959c98dba71d8d78"
mode: GRACEFUL
new_leader_uuid: "e175aaa0dcf3486984c542aaf3563df1"
from {username='slave'} at 127.0.0.1:43578
I20250831 07:57:40.148597 31220 raft_consensus.cc:604] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Received request to transfer leadership to TS e175aaa0dcf3486984c542aaf3563df1
I20250831 07:57:40.153085 31220 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b46e28f6af7247e182075273233a24bc"
dest_uuid: "2961283b97774862959c98dba71d8d78"
mode: GRACEFUL
new_leader_uuid: "e175aaa0dcf3486984c542aaf3563df1"
from {username='slave'} at 127.0.0.1:43578
I20250831 07:57:40.153542 31220 raft_consensus.cc:604] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Received request to transfer leadership to TS e175aaa0dcf3486984c542aaf3563df1
I20250831 07:57:40.157807 31220 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "32f97fdb66fd4c24bcdef839b4c3432a"
dest_uuid: "2961283b97774862959c98dba71d8d78"
mode: GRACEFUL
new_leader_uuid: "e59304d21a1e44febcc404d724fc1577"
from {username='slave'} at 127.0.0.1:43578
I20250831 07:57:40.158246 31220 raft_consensus.cc:604] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Received request to transfer leadership to TS e59304d21a1e44febcc404d724fc1577
I20250831 07:57:40.195048 31452 consensus_queue.cc:1035] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:40.201035 31485 raft_consensus.cc:991] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577: : Instructing follower 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:40.201763 31418 raft_consensus.cc:1079] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Signalling peer 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:40.204490 31221 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d9002d7d65a14e49ad6c842c00ced7f8"
dest_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:43576
I20250831 07:57:40.205360 31221 raft_consensus.cc:491] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:40.205752 31221 raft_consensus.cc:3058] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.208806 31485 consensus_queue.cc:1035] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:40.209105 31492 consensus_queue.cc:1035] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:40.212657 31221 raft_consensus.cc:513] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.217260 31221 leader_election.cc:290] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 2 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:40.217322 31294 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9002d7d65a14e49ad6c842c00ced7f8" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:40.218919 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9002d7d65a14e49ad6c842c00ced7f8" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:40.219576 31370 raft_consensus.cc:3053] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:40.219951 31370 raft_consensus.cc:738] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:40.220834 31370 consensus_queue.cc:260] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.221781 31370 raft_consensus.cc:3058] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.229578 31439 consensus_queue.cc:1035] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:40.229872 31452 consensus_queue.cc:1035] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:40.231640 31370 raft_consensus.cc:2466] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 2.
I20250831 07:57:40.233301 31184 leader_election.cc:304] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577; no voters: e175aaa0dcf3486984c542aaf3563df1
I20250831 07:57:40.234551 31439 raft_consensus.cc:2802] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:40.235244 31439 raft_consensus.cc:695] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Becoming Leader. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:40.236137 31439 consensus_queue.cc:237] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.247988 31485 raft_consensus.cc:991] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577: : Instructing follower 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:40.248433 31485 raft_consensus.cc:1079] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Signalling peer 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:40.249183 31438 consensus_queue.cc:1035] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:40.250932 31221 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f"
dest_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:43576
I20250831 07:57:40.251579 31221 raft_consensus.cc:491] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:40.252045 31221 raft_consensus.cc:3058] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.256162 31116 catalog_manager.cc:5582] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 reported cstate change: term changed from 1 to 2, leader changed from e59304d21a1e44febcc404d724fc1577 (127.29.217.3) to 2961283b97774862959c98dba71d8d78 (127.29.217.1). New cstate: current_term: 2 leader_uuid: "2961283b97774862959c98dba71d8d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:40.259862 31438 raft_consensus.cc:991] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78: : Instructing follower e59304d21a1e44febcc404d724fc1577 to start an election
I20250831 07:57:40.260015 31221 raft_consensus.cc:513] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.260517 31497 raft_consensus.cc:1079] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Signalling peer e59304d21a1e44febcc404d724fc1577 to start an election
I20250831 07:57:40.262583 31294 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:40.262724 31370 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "32f97fdb66fd4c24bcdef839b4c3432a"
dest_uuid: "e59304d21a1e44febcc404d724fc1577"
from {username='slave'} at 127.0.0.1:38610
I20250831 07:57:40.262811 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:40.263222 31294 raft_consensus.cc:3058] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.263489 31369 raft_consensus.cc:3053] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:40.263535 31370 raft_consensus.cc:491] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:40.263985 31369 raft_consensus.cc:738] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:40.264350 31370 raft_consensus.cc:3058] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.264828 31369 consensus_queue.cc:260] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.266129 31369 raft_consensus.cc:3058] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.271359 31294 raft_consensus.cc:2466] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 2.
I20250831 07:57:40.272567 31185 leader_election.cc:304] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1; no voters:
I20250831 07:57:40.262219 31221 leader_election.cc:290] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 2 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:40.273540 31497 raft_consensus.cc:2802] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:40.273702 31369 raft_consensus.cc:2466] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 2.
I20250831 07:57:40.271962 31370 raft_consensus.cc:513] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.275072 31497 raft_consensus.cc:695] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Becoming Leader. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:40.275956 31497 consensus_queue.cc:237] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.276661 31370 leader_election.cc:290] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 2 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001)
I20250831 07:57:40.278128 31294 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32f97fdb66fd4c24bcdef839b4c3432a" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:40.278164 31221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32f97fdb66fd4c24bcdef839b4c3432a" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:40.279258 31221 raft_consensus.cc:3053] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:40.279646 31221 raft_consensus.cc:738] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:40.278744 31294 raft_consensus.cc:3058] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.280687 31221 consensus_queue.cc:260] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.282234 31221 raft_consensus.cc:3058] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.284686 31118 catalog_manager.cc:5582] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 reported cstate change: term changed from 1 to 2, leader changed from e59304d21a1e44febcc404d724fc1577 (127.29.217.3) to 2961283b97774862959c98dba71d8d78 (127.29.217.1). New cstate: current_term: 2 leader_uuid: "2961283b97774862959c98dba71d8d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:40.287075 31221 raft_consensus.cc:2466] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 2.
I20250831 07:57:40.287093 31294 raft_consensus.cc:2466] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 2.
I20250831 07:57:40.288134 31334 leader_election.cc:304] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:40.288797 31452 raft_consensus.cc:2802] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:40.289266 31452 raft_consensus.cc:695] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [term 2 LEADER]: Becoming Leader. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:40.290155 31452 consensus_queue.cc:237] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.296497 31118 catalog_manager.cc:5582] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 reported cstate change: term changed from 1 to 2, leader changed from 2961283b97774862959c98dba71d8d78 (127.29.217.1) to e59304d21a1e44febcc404d724fc1577 (127.29.217.3). New cstate: current_term: 2 leader_uuid: "e59304d21a1e44febcc404d724fc1577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:40.385913 31439 raft_consensus.cc:991] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78: : Instructing follower e175aaa0dcf3486984c542aaf3563df1 to start an election
I20250831 07:57:40.386348 31497 raft_consensus.cc:1079] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Signalling peer e175aaa0dcf3486984c542aaf3563df1 to start an election
I20250831 07:57:40.387853 31294 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c"
dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
from {username='slave'} at 127.0.0.1:39492
I20250831 07:57:40.388304 31294 raft_consensus.cc:491] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:40.388581 31294 raft_consensus.cc:3058] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.393151 31294 raft_consensus.cc:513] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.394518 31294 leader_election.cc:290] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 2 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:40.395408 31221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:40.395571 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:40.396039 31221 raft_consensus.cc:3053] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:40.396145 31370 raft_consensus.cc:3058] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.396428 31221 raft_consensus.cc:738] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:40.397177 31221 consensus_queue.cc:260] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.398386 31221 raft_consensus.cc:3058] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.402684 31370 raft_consensus.cc:2466] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 2.
I20250831 07:57:40.403679 31259 leader_election.cc:304] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:40.404389 31476 raft_consensus.cc:2802] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:40.404309 31221 raft_consensus.cc:2466] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 2.
I20250831 07:57:40.404848 31476 raft_consensus.cc:695] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Becoming Leader. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Running, Role: LEADER
I20250831 07:57:40.405660 31476 consensus_queue.cc:237] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.411775 31118 catalog_manager.cc:5582] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 reported cstate change: term changed from 1 to 2, leader changed from 2961283b97774862959c98dba71d8d78 (127.29.217.1) to e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "e175aaa0dcf3486984c542aaf3563df1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:40.422572 31497 consensus_queue.cc:1035] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:40.432121 31416 consensus_queue.cc:1035] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:40.446959 31438 raft_consensus.cc:991] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78: : Instructing follower e175aaa0dcf3486984c542aaf3563df1 to start an election
I20250831 07:57:40.447011 31452 consensus_queue.cc:1035] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:40.447623 31416 raft_consensus.cc:1079] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Signalling peer e175aaa0dcf3486984c542aaf3563df1 to start an election
I20250831 07:57:40.449143 31294 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b46e28f6af7247e182075273233a24bc"
dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
from {username='slave'} at 127.0.0.1:39492
I20250831 07:57:40.449667 31294 raft_consensus.cc:491] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:40.450021 31294 raft_consensus.cc:3058] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.457572 31294 raft_consensus.cc:513] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.459372 31485 consensus_queue.cc:1035] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:57:40.460194 31294 leader_election.cc:290] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 2 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:40.460126 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b46e28f6af7247e182075273233a24bc" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:40.460898 31220 raft_consensus.cc:3053] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:40.461210 31220 raft_consensus.cc:738] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:40.461498 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b46e28f6af7247e182075273233a24bc" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:40.462054 31370 raft_consensus.cc:3058] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.461974 31220 consensus_queue.cc:260] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.462950 31220 raft_consensus.cc:3058] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.469776 31370 raft_consensus.cc:2466] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 2.
I20250831 07:57:40.469985 31485 raft_consensus.cc:991] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577: : Instructing follower e175aaa0dcf3486984c542aaf3563df1 to start an election
I20250831 07:57:40.470590 31485 raft_consensus.cc:1079] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Signalling peer e175aaa0dcf3486984c542aaf3563df1 to start an election
I20250831 07:57:40.471071 31259 leader_election.cc:304] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:40.471750 31476 raft_consensus.cc:2802] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:40.472218 31476 raft_consensus.cc:695] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Becoming Leader. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Running, Role: LEADER
I20250831 07:57:40.472955 31476 consensus_queue.cc:237] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.475224 31220 raft_consensus.cc:2466] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 2.
I20250831 07:57:40.477614 31294 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5"
dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
from {username='slave'} at 127.0.0.1:39498
I20250831 07:57:40.479060 31294 raft_consensus.cc:491] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:40.479357 31294 raft_consensus.cc:3058] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.480949 31417 raft_consensus.cc:991] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1: : Instructing follower 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:40.480453 31116 catalog_manager.cc:5582] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 reported cstate change: term changed from 1 to 2, leader changed from 2961283b97774862959c98dba71d8d78 (127.29.217.1) to e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "e175aaa0dcf3486984c542aaf3563df1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:40.481376 31476 raft_consensus.cc:1079] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 1 LEADER]: Signalling peer 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:40.484648 31220 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a"
dest_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:43570
I20250831 07:57:40.484887 31294 raft_consensus.cc:513] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.485175 31220 raft_consensus.cc:491] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:40.485589 31220 raft_consensus.cc:3058] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.486521 31294 leader_election.cc:290] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 2 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:40.488636 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:40.488794 31219 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:40.489320 31370 raft_consensus.cc:3053] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:40.489452 31219 raft_consensus.cc:3058] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.489710 31370 raft_consensus.cc:738] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:40.490680 31370 consensus_queue.cc:260] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.491957 31370 raft_consensus.cc:3058] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.492977 31220 raft_consensus.cc:513] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.495589 31294 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:40.496155 31294 raft_consensus.cc:3053] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:40.496460 31294 raft_consensus.cc:738] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Running, Role: LEADER
I20250831 07:57:40.497236 31219 raft_consensus.cc:2466] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 2.
I20250831 07:57:40.497228 31294 consensus_queue.cc:260] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.498399 31294 raft_consensus.cc:3058] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.498456 31260 leader_election.cc:304] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1; no voters:
I20250831 07:57:40.498564 31370 raft_consensus.cc:2466] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 2.
I20250831 07:57:40.499356 31476 raft_consensus.cc:2802] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:40.499851 31476 raft_consensus.cc:695] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Becoming Leader. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Running, Role: LEADER
I20250831 07:57:40.500633 31476 consensus_queue.cc:237] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.503466 31220 leader_election.cc:290] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 2 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:40.505066 31294 raft_consensus.cc:2466] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 2.
I20250831 07:57:40.505172 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:40.505828 31370 raft_consensus.cc:3058] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.505980 31185 leader_election.cc:304] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1; no voters:
I20250831 07:57:40.506757 31416 raft_consensus.cc:2802] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:40.507184 31416 raft_consensus.cc:695] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Becoming Leader. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:40.508009 31416 consensus_queue.cc:237] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.508983 31116 catalog_manager.cc:5582] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 reported cstate change: term changed from 1 to 2, leader changed from e59304d21a1e44febcc404d724fc1577 (127.29.217.3) to e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "e175aaa0dcf3486984c542aaf3563df1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:40.511528 31370 raft_consensus.cc:2466] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 2.
I20250831 07:57:40.515480 31118 catalog_manager.cc:5582] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 reported cstate change: term changed from 1 to 2, leader changed from e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2) to 2961283b97774862959c98dba71d8d78 (127.29.217.1). New cstate: current_term: 2 leader_uuid: "2961283b97774862959c98dba71d8d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:40.516284 31417 raft_consensus.cc:991] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1: : Instructing follower 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:40.516763 31417 raft_consensus.cc:1079] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 1 LEADER]: Signalling peer 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:40.518249 31220 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e2415c634a154f269bf1f47ba35a97c3"
dest_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:43570
I20250831 07:57:40.518775 31220 raft_consensus.cc:491] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:40.519133 31220 raft_consensus.cc:3058] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.524341 31220 raft_consensus.cc:513] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.525941 31220 leader_election.cc:290] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 2 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:40.526857 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2415c634a154f269bf1f47ba35a97c3" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:40.526924 31294 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2415c634a154f269bf1f47ba35a97c3" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:40.527355 31370 raft_consensus.cc:3058] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.527465 31294 raft_consensus.cc:3053] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:40.527777 31294 raft_consensus.cc:738] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Running, Role: LEADER
I20250831 07:57:40.528436 31294 consensus_queue.cc:260] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.529265 31294 raft_consensus.cc:3058] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.532215 31370 raft_consensus.cc:2466] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 2.
I20250831 07:57:40.533157 31184 leader_election.cc:304] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:40.533931 31416 raft_consensus.cc:2802] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:40.534269 31294 raft_consensus.cc:2466] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 2.
I20250831 07:57:40.534292 31416 raft_consensus.cc:695] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Becoming Leader. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:40.535272 31416 consensus_queue.cc:237] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.541143 31118 catalog_manager.cc:5582] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 reported cstate change: term changed from 1 to 2, leader changed from e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2) to 2961283b97774862959c98dba71d8d78 (127.29.217.1). New cstate: current_term: 2 leader_uuid: "2961283b97774862959c98dba71d8d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:40.588852 31485 raft_consensus.cc:991] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577: : Instructing follower e175aaa0dcf3486984c542aaf3563df1 to start an election
I20250831 07:57:40.589362 31492 raft_consensus.cc:1079] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Signalling peer e175aaa0dcf3486984c542aaf3563df1 to start an election
I20250831 07:57:40.590896 31294 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa"
dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
from {username='slave'} at 127.0.0.1:39498
I20250831 07:57:40.591342 31294 raft_consensus.cc:491] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:40.591622 31294 raft_consensus.cc:3058] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.596084 31294 raft_consensus.cc:513] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.597733 31294 leader_election.cc:290] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 2 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:40.598575 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:40.598697 31370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:40.599053 31220 raft_consensus.cc:3058] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.599148 31370 raft_consensus.cc:3053] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:40.599444 31370 raft_consensus.cc:738] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:40.600400 31370 consensus_queue.cc:260] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.601473 31370 raft_consensus.cc:3058] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.604784 31220 raft_consensus.cc:2466] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 2.
I20250831 07:57:40.605684 31260 leader_election.cc:304] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1; no voters:
I20250831 07:57:40.606324 31417 raft_consensus.cc:2802] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:40.607226 31417 raft_consensus.cc:695] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Becoming Leader. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Running, Role: LEADER
I20250831 07:57:40.607247 31370 raft_consensus.cc:2466] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 2.
I20250831 07:57:40.608173 31417 consensus_queue.cc:237] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:40.614532 31118 catalog_manager.cc:5582] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 reported cstate change: term changed from 1 to 2, leader changed from e59304d21a1e44febcc404d724fc1577 (127.29.217.3) to e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "e175aaa0dcf3486984c542aaf3563df1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:40.724479 31294 raft_consensus.cc:3058] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:40.729399 31294 raft_consensus.cc:1273] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.730665 31416 consensus_queue.cc:1035] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:40.731392 31294 raft_consensus.cc:1273] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Refusing update from remote peer e59304d21a1e44febcc404d724fc1577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.732617 31492 consensus_queue.cc:1035] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:40.738807 31370 raft_consensus.cc:1273] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.744676 31497 consensus_queue.cc:1035] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:40.747515 31220 raft_consensus.cc:1273] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Refusing update from remote peer e59304d21a1e44febcc404d724fc1577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.748778 31492 consensus_queue.cc:1035] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:40.817342 31370 raft_consensus.cc:1273] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.818514 31497 consensus_queue.cc:1035] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:40.825177 31294 raft_consensus.cc:1273] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.826462 31497 consensus_queue.cc:1035] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:40.850262 31220 raft_consensus.cc:1273] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Refusing update from remote peer e175aaa0dcf3486984c542aaf3563df1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.851516 31417 consensus_queue.cc:1035] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:40.859408 31370 raft_consensus.cc:1273] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Refusing update from remote peer e175aaa0dcf3486984c542aaf3563df1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.860993 31448 consensus_queue.cc:1035] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:40.888032 31220 raft_consensus.cc:1273] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Refusing update from remote peer e175aaa0dcf3486984c542aaf3563df1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.889197 31448 consensus_queue.cc:1035] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:40.896296 31370 raft_consensus.cc:1273] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Refusing update from remote peer e175aaa0dcf3486984c542aaf3563df1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.897671 31476 consensus_queue.cc:1035] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:40.940552 31370 raft_consensus.cc:1273] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Refusing update from remote peer e175aaa0dcf3486984c542aaf3563df1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.941504 31476 consensus_queue.cc:1035] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:40.948989 31220 raft_consensus.cc:1273] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Refusing update from remote peer e175aaa0dcf3486984c542aaf3563df1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.950284 31417 consensus_queue.cc:1035] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:40.973659 31370 raft_consensus.cc:1273] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.974762 31416 consensus_queue.cc:1035] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:40.982385 31294 raft_consensus.cc:1273] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:40.983733 31497 consensus_queue.cc:1035] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:41.073796 31370 raft_consensus.cc:1273] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Refusing update from remote peer e175aaa0dcf3486984c542aaf3563df1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:41.074878 31417 consensus_queue.cc:1035] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:41.082561 31220 raft_consensus.cc:1273] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Refusing update from remote peer e175aaa0dcf3486984c542aaf3563df1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:41.084293 31476 consensus_queue.cc:1035] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:41.116199 31294 raft_consensus.cc:1273] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:41.117586 31497 consensus_queue.cc:1035] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:41.122438 31370 raft_consensus.cc:1273] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:41.123875 31438 consensus_queue.cc:1035] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:43.188581 30564 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20250831 07:57:43.189188 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:43.204826 31220 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a"
dest_uuid: "2961283b97774862959c98dba71d8d78"
mode: GRACEFUL
new_leader_uuid: "e59304d21a1e44febcc404d724fc1577"
from {username='slave'} at 127.0.0.1:43582
I20250831 07:57:43.205293 31220 raft_consensus.cc:604] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Received request to transfer leadership to TS e59304d21a1e44febcc404d724fc1577
I20250831 07:57:43.207369 31220 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf"
dest_uuid: "2961283b97774862959c98dba71d8d78"
mode: GRACEFUL
new_leader_uuid: "e59304d21a1e44febcc404d724fc1577"
from {username='slave'} at 127.0.0.1:43582
I20250831 07:57:43.207844 31220 raft_consensus.cc:604] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Received request to transfer leadership to TS e59304d21a1e44febcc404d724fc1577
I20250831 07:57:43.208762 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250831 07:57:43.209326 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:43.437922 31514 raft_consensus.cc:991] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78: : Instructing follower e59304d21a1e44febcc404d724fc1577 to start an election
I20250831 07:57:43.438341 31511 raft_consensus.cc:1079] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Signalling peer e59304d21a1e44febcc404d724fc1577 to start an election
I20250831 07:57:43.439736 31370 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf"
dest_uuid: "e59304d21a1e44febcc404d724fc1577"
from {username='slave'} at 127.0.0.1:38610
I20250831 07:57:43.440184 31370 raft_consensus.cc:491] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:43.440425 31370 raft_consensus.cc:3058] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:43.444986 31370 raft_consensus.cc:513] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:43.446623 31370 leader_election.cc:290] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 2 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001)
I20250831 07:57:43.447531 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:43.447803 31295 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:43.448215 31220 raft_consensus.cc:3053] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:57:43.448347 31295 raft_consensus.cc:3058] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:43.448577 31220 raft_consensus.cc:738] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:43.449373 31220 consensus_queue.cc:260] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:43.450572 31220 raft_consensus.cc:3058] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:57:43.455235 31295 raft_consensus.cc:2466] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 2.
I20250831 07:57:43.456734 31334 leader_election.cc:304] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:43.457226 31220 raft_consensus.cc:2466] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 2.
I20250831 07:57:43.457419 31492 raft_consensus.cc:2802] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:57:43.458420 31492 raft_consensus.cc:695] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 2 LEADER]: Becoming Leader. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:43.459146 31492 consensus_queue.cc:237] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:43.465119 31118 catalog_manager.cc:5582] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 reported cstate change: term changed from 1 to 2, leader changed from 2961283b97774862959c98dba71d8d78 (127.29.217.1) to e59304d21a1e44febcc404d724fc1577 (127.29.217.3). New cstate: current_term: 2 leader_uuid: "e59304d21a1e44febcc404d724fc1577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:43.767403 31514 raft_consensus.cc:991] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78: : Instructing follower e59304d21a1e44febcc404d724fc1577 to start an election
I20250831 07:57:43.767845 31514 raft_consensus.cc:1079] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Signalling peer e59304d21a1e44febcc404d724fc1577 to start an election
I20250831 07:57:43.769136 31370 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a"
dest_uuid: "e59304d21a1e44febcc404d724fc1577"
from {username='slave'} at 127.0.0.1:38610
I20250831 07:57:43.769546 31370 raft_consensus.cc:491] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:43.769791 31370 raft_consensus.cc:3058] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:43.773789 31370 raft_consensus.cc:513] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:43.774998 31370 leader_election.cc:290] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 3 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001)
I20250831 07:57:43.776120 31295 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:43.776042 31220 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:43.776729 31295 raft_consensus.cc:3058] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:43.776778 31220 raft_consensus.cc:3053] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:57:43.777201 31220 raft_consensus.cc:738] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:43.777978 31220 consensus_queue.cc:260] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:43.779165 31220 raft_consensus.cc:3058] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:43.783366 31295 raft_consensus.cc:2466] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 3.
I20250831 07:57:43.784492 31334 leader_election.cc:304] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:43.785248 31492 raft_consensus.cc:2802] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:57:43.785486 31220 raft_consensus.cc:2466] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 3.
I20250831 07:57:43.785678 31492 raft_consensus.cc:695] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 3 LEADER]: Becoming Leader. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:43.786412 31492 consensus_queue.cc:237] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:43.794497 31118 catalog_manager.cc:5582] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 reported cstate change: term changed from 2 to 3, leader changed from 2961283b97774862959c98dba71d8d78 (127.29.217.1) to e59304d21a1e44febcc404d724fc1577 (127.29.217.3). New cstate: current_term: 3 leader_uuid: "e59304d21a1e44febcc404d724fc1577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:43.850273 31295 raft_consensus.cc:1273] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Refusing update from remote peer e59304d21a1e44febcc404d724fc1577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:43.851425 31492 consensus_queue.cc:1035] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:57:43.860415 31220 raft_consensus.cc:1273] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Refusing update from remote peer e59304d21a1e44febcc404d724fc1577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:57:43.862866 31492 consensus_queue.cc:1035] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:57:44.210374 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:44.216461 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:44.216894 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:44.224933 31220 raft_consensus.cc:1273] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Refusing update from remote peer e59304d21a1e44febcc404d724fc1577: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:44.226107 31525 consensus_queue.cc:1035] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:44.237411 31294 raft_consensus.cc:1273] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Refusing update from remote peer e59304d21a1e44febcc404d724fc1577: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:44.238981 31529 consensus_queue.cc:1035] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:45.217726 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:45.223528 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:45.224045 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:46.224998 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:46.231755 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:46.232197 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:47.233310 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:47.239775 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:47.240249 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:48.241286 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:48.246421 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:48.246838 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:49.247903 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:49.252871 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:49.253293 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:50.254243 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:50.259572 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:50.260020 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:51.260898 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:51.266237 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:51.266677 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:52.267519 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:52.272766 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:52.273196 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:53.301575 30564 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20250831 07:57:53.310164 31369 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf"
dest_uuid: "e59304d21a1e44febcc404d724fc1577"
mode: GRACEFUL
new_leader_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:38624
I20250831 07:57:53.310923 31369 raft_consensus.cc:604] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 2 LEADER]: Received request to transfer leadership to TS 2961283b97774862959c98dba71d8d78
I20250831 07:57:53.317885 31369 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a"
dest_uuid: "e59304d21a1e44febcc404d724fc1577"
mode: GRACEFUL
new_leader_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:38624
I20250831 07:57:53.318383 31369 raft_consensus.cc:604] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 3 LEADER]: Received request to transfer leadership to TS 2961283b97774862959c98dba71d8d78
I20250831 07:57:53.323278 31295 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5"
dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
mode: GRACEFUL
new_leader_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:39510
I20250831 07:57:53.323952 31295 raft_consensus.cc:604] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Received request to transfer leadership to TS 2961283b97774862959c98dba71d8d78
I20250831 07:57:53.328634 31295 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa"
dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
mode: GRACEFUL
new_leader_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:39510
I20250831 07:57:53.329149 31295 raft_consensus.cc:604] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Received request to transfer leadership to TS 2961283b97774862959c98dba71d8d78
I20250831 07:57:53.333253 31295 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c"
dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
mode: GRACEFUL
new_leader_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:39510
I20250831 07:57:53.333676 31295 raft_consensus.cc:604] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Received request to transfer leadership to TS 2961283b97774862959c98dba71d8d78
I20250831 07:57:53.421058 31541 raft_consensus.cc:991] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577: : Instructing follower 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:53.421540 31538 raft_consensus.cc:1079] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 2 LEADER]: Signalling peer 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:53.422931 31219 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf"
dest_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:43576
I20250831 07:57:53.423362 31219 raft_consensus.cc:491] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:53.423674 31219 raft_consensus.cc:3058] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:53.429072 31219 raft_consensus.cc:513] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.430764 31219 leader_election.cc:290] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 3 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:53.431701 31295 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:53.431890 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:53.432462 31295 raft_consensus.cc:3058] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:53.432545 31369 raft_consensus.cc:3053] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:57:53.432925 31369 raft_consensus.cc:738] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:53.433574 31369 consensus_queue.cc:260] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.434530 31369 raft_consensus.cc:3058] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:53.437319 31295 raft_consensus.cc:2466] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 3.
I20250831 07:57:53.438426 31185 leader_election.cc:304] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1; no voters:
I20250831 07:57:53.439328 31514 raft_consensus.cc:2802] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:57:53.440393 31514 raft_consensus.cc:695] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Becoming Leader. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:53.440907 31369 raft_consensus.cc:2466] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 3.
I20250831 07:57:53.441399 31514 consensus_queue.cc:237] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.448441 31116 catalog_manager.cc:5582] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 reported cstate change: term changed from 2 to 3, leader changed from e59304d21a1e44febcc404d724fc1577 (127.29.217.3) to 2961283b97774862959c98dba71d8d78 (127.29.217.1). New cstate: current_term: 3 leader_uuid: "2961283b97774862959c98dba71d8d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:53.536707 31535 raft_consensus.cc:991] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1: : Instructing follower 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:53.537225 31540 raft_consensus.cc:1079] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Signalling peer 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:53.538578 31219 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5"
dest_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:43570
I20250831 07:57:53.539053 31219 raft_consensus.cc:491] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:53.539317 31219 raft_consensus.cc:3058] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:53.544065 31219 raft_consensus.cc:513] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.545626 31219 leader_election.cc:290] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 3 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:53.546666 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:53.546666 31295 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:53.547322 31369 raft_consensus.cc:3058] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:53.547367 31295 raft_consensus.cc:3053] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:57:53.547858 31295 raft_consensus.cc:738] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Running, Role: LEADER
I20250831 07:57:53.548673 31295 consensus_queue.cc:260] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.549953 31295 raft_consensus.cc:3058] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:53.554432 31369 raft_consensus.cc:2466] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 3.
I20250831 07:57:53.555724 31184 leader_election.cc:304] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:53.556095 31295 raft_consensus.cc:2466] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 3.
I20250831 07:57:53.556564 31514 raft_consensus.cc:2802] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:57:53.557113 31514 raft_consensus.cc:695] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Becoming Leader. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:53.557816 31514 consensus_queue.cc:237] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.564224 31116 catalog_manager.cc:5582] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 reported cstate change: term changed from 2 to 3, leader changed from e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2) to 2961283b97774862959c98dba71d8d78 (127.29.217.1). New cstate: current_term: 3 leader_uuid: "2961283b97774862959c98dba71d8d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:53.656704 31535 raft_consensus.cc:991] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1: : Instructing follower 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:53.657150 31535 raft_consensus.cc:1079] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Signalling peer 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:53.658334 31219 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c"
dest_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:43570
I20250831 07:57:53.658725 31219 raft_consensus.cc:491] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:53.658972 31219 raft_consensus.cc:3058] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:53.663527 31219 raft_consensus.cc:513] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.665042 31219 leader_election.cc:290] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 3 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:53.665714 31295 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:53.665983 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:53.666225 31295 raft_consensus.cc:3053] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:57:53.666541 31369 raft_consensus.cc:3058] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:53.666594 31295 raft_consensus.cc:738] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Running, Role: LEADER
I20250831 07:57:53.667280 31295 consensus_queue.cc:260] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.668205 31295 raft_consensus.cc:3058] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:53.671602 31540 raft_consensus.cc:991] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1: : Instructing follower 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:53.672096 31535 raft_consensus.cc:1079] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Signalling peer 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:53.672140 31369 raft_consensus.cc:2466] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 3.
I20250831 07:57:53.673141 31184 leader_election.cc:304] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:53.673527 31295 raft_consensus.cc:2466] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 3.
I20250831 07:57:53.673517 31219 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa"
dest_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:43570
I20250831 07:57:53.673941 31514 raft_consensus.cc:2802] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:57:53.674245 31219 raft_consensus.cc:491] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:53.674615 31219 raft_consensus.cc:3058] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:53.674959 31514 raft_consensus.cc:695] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Becoming Leader. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:53.675554 31514 consensus_queue.cc:237] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.679845 31219 raft_consensus.cc:513] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.681587 31219 leader_election.cc:290] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 3 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:53.682525 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:53.683034 31369 raft_consensus.cc:3058] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:53.682963 31116 catalog_manager.cc:5582] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 reported cstate change: term changed from 2 to 3, leader changed from e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2) to 2961283b97774862959c98dba71d8d78 (127.29.217.1). New cstate: current_term: 3 leader_uuid: "2961283b97774862959c98dba71d8d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:53.688249 31369 raft_consensus.cc:2466] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 3.
I20250831 07:57:53.689215 31184 leader_election.cc:304] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:53.689991 31295 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9ad3c047ea14d648689485283a199aa" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:53.690515 31514 raft_consensus.cc:2802] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:57:53.691231 31295 raft_consensus.cc:3053] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:57:53.691615 31295 raft_consensus.cc:738] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Running, Role: LEADER
I20250831 07:57:53.692669 31295 consensus_queue.cc:260] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.693921 31295 raft_consensus.cc:3058] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:53.702186 31295 raft_consensus.cc:2466] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 3.
I20250831 07:57:53.703001 31514 raft_consensus.cc:695] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Becoming Leader. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:53.703776 31514 consensus_queue.cc:237] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.709968 31116 catalog_manager.cc:5582] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 reported cstate change: term changed from 2 to 3, leader changed from e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2) to 2961283b97774862959c98dba71d8d78 (127.29.217.1). New cstate: current_term: 3 leader_uuid: "2961283b97774862959c98dba71d8d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:53.783151 31541 raft_consensus.cc:991] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577: : Instructing follower 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:53.783572 31541 raft_consensus.cc:1079] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 3 LEADER]: Signalling peer 2961283b97774862959c98dba71d8d78 to start an election
I20250831 07:57:53.784811 31219 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a"
dest_uuid: "2961283b97774862959c98dba71d8d78"
from {username='slave'} at 127.0.0.1:43576
I20250831 07:57:53.785239 31219 raft_consensus.cc:491] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:53.785508 31219 raft_consensus.cc:3058] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:53.790058 31219 raft_consensus.cc:513] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.791594 31219 leader_election.cc:290] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 4 election: Requested vote from peers e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:53.792284 31295 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:53.792537 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3db868229f2e47d0af6034b101d57e7a" candidate_uuid: "2961283b97774862959c98dba71d8d78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:53.792837 31295 raft_consensus.cc:3058] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:53.793046 31369 raft_consensus.cc:3053] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:57:53.793378 31369 raft_consensus.cc:738] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:53.794157 31369 consensus_queue.cc:260] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.795338 31369 raft_consensus.cc:3058] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:53.799176 31295 raft_consensus.cc:2466] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 4.
I20250831 07:57:53.800184 31185 leader_election.cc:304] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2961283b97774862959c98dba71d8d78, e175aaa0dcf3486984c542aaf3563df1; no voters:
I20250831 07:57:53.800853 31514 raft_consensus.cc:2802] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:57:53.801263 31514 raft_consensus.cc:695] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 4 LEADER]: Becoming Leader. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:53.801601 31369 raft_consensus.cc:2466] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2961283b97774862959c98dba71d8d78 in term 4.
I20250831 07:57:53.802078 31514 consensus_queue.cc:237] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:53.808596 31116 catalog_manager.cc:5582] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 reported cstate change: term changed from 3 to 4, leader changed from e59304d21a1e44febcc404d724fc1577 (127.29.217.3) to 2961283b97774862959c98dba71d8d78 (127.29.217.1). New cstate: current_term: 4 leader_uuid: "2961283b97774862959c98dba71d8d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:53.936805 31295 raft_consensus.cc:1273] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:53.938072 31514 consensus_queue.cc:1035] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:53.947367 31369 raft_consensus.cc:1273] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:53.948864 31550 consensus_queue.cc:1035] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:57:54.043145 31295 raft_consensus.cc:1273] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:54.044129 31550 consensus_queue.cc:1035] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:54.052177 31369 raft_consensus.cc:1273] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:54.053359 31514 consensus_queue.cc:1035] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:54.092140 31295 raft_consensus.cc:1273] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:54.093230 31514 consensus_queue.cc:1035] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:54.100485 31369 raft_consensus.cc:1273] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:54.102244 31514 consensus_queue.cc:1035] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:54.119774 31369 raft_consensus.cc:1273] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:54.120800 31514 consensus_queue.cc:1035] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:54.129199 31295 raft_consensus.cc:1273] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:54.130465 31514 consensus_queue.cc:1035] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:54.262195 31369 raft_consensus.cc:1273] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:54.263340 31550 consensus_queue.cc:1035] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:57:54.271615 31295 raft_consensus.cc:1273] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Refusing update from remote peer 2961283b97774862959c98dba71d8d78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:54.273685 31550 consensus_queue.cc:1035] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:57:56.367650 30564 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20250831 07:57:56.368279 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:56.375936 31219 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c"
dest_uuid: "2961283b97774862959c98dba71d8d78"
mode: GRACEFUL
new_leader_uuid: "e175aaa0dcf3486984c542aaf3563df1"
from {username='slave'} at 127.0.0.1:43582
I20250831 07:57:56.376434 31219 raft_consensus.cc:604] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Received request to transfer leadership to TS e175aaa0dcf3486984c542aaf3563df1
I20250831 07:57:56.378788 31219 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f"
dest_uuid: "2961283b97774862959c98dba71d8d78"
mode: GRACEFUL
new_leader_uuid: "e59304d21a1e44febcc404d724fc1577"
from {username='slave'} at 127.0.0.1:43582
I20250831 07:57:56.379276 31219 raft_consensus.cc:604] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Received request to transfer leadership to TS e59304d21a1e44febcc404d724fc1577
I20250831 07:57:56.381516 31219 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5"
dest_uuid: "2961283b97774862959c98dba71d8d78"
mode: GRACEFUL
new_leader_uuid: "e59304d21a1e44febcc404d724fc1577"
from {username='slave'} at 127.0.0.1:43582
I20250831 07:57:56.382246 31550 raft_consensus.cc:991] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78: : Instructing follower e59304d21a1e44febcc404d724fc1577 to start an election
I20250831 07:57:56.382560 31219 raft_consensus.cc:604] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Received request to transfer leadership to TS e59304d21a1e44febcc404d724fc1577
I20250831 07:57:56.382776 31566 raft_consensus.cc:1079] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Signalling peer e59304d21a1e44febcc404d724fc1577 to start an election
I20250831 07:57:56.384289 31369 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f"
dest_uuid: "e59304d21a1e44febcc404d724fc1577"
from {username='slave'} at 127.0.0.1:38610
I20250831 07:57:56.384713 31369 raft_consensus.cc:491] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:56.385051 31369 raft_consensus.cc:3058] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:56.385282 31219 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf"
dest_uuid: "2961283b97774862959c98dba71d8d78"
mode: GRACEFUL
new_leader_uuid: "e175aaa0dcf3486984c542aaf3563df1"
from {username='slave'} at 127.0.0.1:43582
I20250831 07:57:56.385812 31219 raft_consensus.cc:604] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Received request to transfer leadership to TS e175aaa0dcf3486984c542aaf3563df1
I20250831 07:57:56.386700 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250831 07:57:56.387110 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:56.390276 31369 raft_consensus.cc:513] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:56.392078 31369 leader_election.cc:290] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 3 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001)
I20250831 07:57:56.392863 31219 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:56.393090 31295 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7800f3d155894012ab8b8a3067479d5f" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:56.393440 31219 raft_consensus.cc:3053] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:57:56.393539 31295 raft_consensus.cc:3058] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:56.393790 31219 raft_consensus.cc:738] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:56.394513 31219 consensus_queue.cc:260] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:56.395649 31219 raft_consensus.cc:3058] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:57:56.399784 31295 raft_consensus.cc:2466] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 3.
I20250831 07:57:56.400907 31334 leader_election.cc:304] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:56.401566 31538 raft_consensus.cc:2802] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:57:56.401813 31219 raft_consensus.cc:2466] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 3.
I20250831 07:57:56.402477 31538 raft_consensus.cc:695] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 3 LEADER]: Becoming Leader. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:56.403360 31538 consensus_queue.cc:237] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:56.409286 31116 catalog_manager.cc:5582] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 reported cstate change: term changed from 2 to 3, leader changed from 2961283b97774862959c98dba71d8d78 (127.29.217.1) to e59304d21a1e44febcc404d724fc1577 (127.29.217.3). New cstate: current_term: 3 leader_uuid: "e59304d21a1e44febcc404d724fc1577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:56.682179 31550 raft_consensus.cc:991] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78: : Instructing follower e175aaa0dcf3486984c542aaf3563df1 to start an election
I20250831 07:57:56.682610 31566 raft_consensus.cc:1079] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Signalling peer e175aaa0dcf3486984c542aaf3563df1 to start an election
I20250831 07:57:56.683956 31294 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf"
dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
from {username='slave'} at 127.0.0.1:39492
I20250831 07:57:56.684420 31294 raft_consensus.cc:491] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:56.684697 31294 raft_consensus.cc:3058] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:56.689720 31294 raft_consensus.cc:513] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:56.691562 31294 leader_election.cc:290] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 4 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:56.692502 31219 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:56.692541 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6698eb6af2f4c888796b8e9f1003baf" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:56.693091 31219 raft_consensus.cc:3053] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:57:56.693152 31369 raft_consensus.cc:3058] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:56.693356 31219 raft_consensus.cc:738] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:56.693975 31219 consensus_queue.cc:260] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:56.694773 31219 raft_consensus.cc:3058] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:56.697319 31369 raft_consensus.cc:2466] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 4.
I20250831 07:57:56.698318 31259 leader_election.cc:304] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:56.698839 31219 raft_consensus.cc:2466] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 4.
I20250831 07:57:56.698944 31535 raft_consensus.cc:2802] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:57:56.700131 31535 raft_consensus.cc:695] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 4 LEADER]: Becoming Leader. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Running, Role: LEADER
I20250831 07:57:56.700758 31535 consensus_queue.cc:237] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:56.706807 31116 catalog_manager.cc:5582] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 reported cstate change: term changed from 3 to 4, leader changed from 2961283b97774862959c98dba71d8d78 (127.29.217.1) to e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2). New cstate: current_term: 4 leader_uuid: "e175aaa0dcf3486984c542aaf3563df1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:56.793183 31550 raft_consensus.cc:991] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78: : Instructing follower e175aaa0dcf3486984c542aaf3563df1 to start an election
I20250831 07:57:56.793576 31566 raft_consensus.cc:1079] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Signalling peer e175aaa0dcf3486984c542aaf3563df1 to start an election
I20250831 07:57:56.794761 31294 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c"
dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
from {username='slave'} at 127.0.0.1:39492
I20250831 07:57:56.795125 31294 raft_consensus.cc:491] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:56.795336 31294 raft_consensus.cc:3058] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:56.799324 31294 raft_consensus.cc:513] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:56.800606 31294 leader_election.cc:290] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 4 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e59304d21a1e44febcc404d724fc1577 (127.29.217.3:36251)
I20250831 07:57:56.801324 31219 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:56.801441 31369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d2c02218be24f76ac0c207781177c0c" candidate_uuid: "e175aaa0dcf3486984c542aaf3563df1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e59304d21a1e44febcc404d724fc1577"
I20250831 07:57:56.801858 31219 raft_consensus.cc:3053] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:57:56.801975 31369 raft_consensus.cc:3058] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:56.802109 31219 raft_consensus.cc:738] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:56.802799 31219 consensus_queue.cc:260] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:56.803567 31219 raft_consensus.cc:3058] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:56.806547 31369 raft_consensus.cc:2466] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 4.
I20250831 07:57:56.807801 31259 leader_election.cc:304] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:56.808359 31219 raft_consensus.cc:2466] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e175aaa0dcf3486984c542aaf3563df1 in term 4.
I20250831 07:57:56.808490 31535 raft_consensus.cc:2802] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:57:56.808919 31535 raft_consensus.cc:695] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 4 LEADER]: Becoming Leader. State: Replica: e175aaa0dcf3486984c542aaf3563df1, State: Running, Role: LEADER
I20250831 07:57:56.809681 31535 consensus_queue.cc:237] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:56.815809 31116 catalog_manager.cc:5582] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 reported cstate change: term changed from 3 to 4, leader changed from 2961283b97774862959c98dba71d8d78 (127.29.217.1) to e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2). New cstate: current_term: 4 leader_uuid: "e175aaa0dcf3486984c542aaf3563df1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20250831 07:57:56.836094 31294 raft_consensus.cc:1273] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Refusing update from remote peer e59304d21a1e44febcc404d724fc1577: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:56.837112 31538 consensus_queue.cc:1035] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:56.845904 31219 raft_consensus.cc:1273] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Refusing update from remote peer e59304d21a1e44febcc404d724fc1577: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:57:56.847355 31538 consensus_queue.cc:1035] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:57:56.925906 31550 raft_consensus.cc:991] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78: : Instructing follower e59304d21a1e44febcc404d724fc1577 to start an election
I20250831 07:57:56.926659 31566 raft_consensus.cc:1079] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Signalling peer e59304d21a1e44febcc404d724fc1577 to start an election
I20250831 07:57:56.928361 31369 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5"
dest_uuid: "e59304d21a1e44febcc404d724fc1577"
from {username='slave'} at 127.0.0.1:38610
I20250831 07:57:56.929028 31369 raft_consensus.cc:491] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:57:56.929431 31369 raft_consensus.cc:3058] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:56.935904 31369 raft_consensus.cc:513] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:56.937434 31369 leader_election.cc:290] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 4 election: Requested vote from peers 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687), e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001)
I20250831 07:57:56.939090 31219 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2961283b97774862959c98dba71d8d78"
I20250831 07:57:56.939401 31294 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acc1b872a4834f68a4494a11d4487df5" candidate_uuid: "e59304d21a1e44febcc404d724fc1577" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e175aaa0dcf3486984c542aaf3563df1"
I20250831 07:57:56.939708 31219 raft_consensus.cc:3053] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:57:56.940063 31219 raft_consensus.cc:738] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2961283b97774862959c98dba71d8d78, State: Running, Role: LEADER
I20250831 07:57:56.940080 31294 raft_consensus.cc:3058] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:56.940810 31219 consensus_queue.cc:260] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:56.942080 31219 raft_consensus.cc:3058] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:57:56.945782 31294 raft_consensus.cc:2466] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 4.
I20250831 07:57:56.946871 31334 leader_election.cc:304] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e175aaa0dcf3486984c542aaf3563df1, e59304d21a1e44febcc404d724fc1577; no voters:
I20250831 07:57:56.947217 31219 raft_consensus.cc:2466] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59304d21a1e44febcc404d724fc1577 in term 4.
I20250831 07:57:56.947978 31538 raft_consensus.cc:2802] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:57:56.948418 31538 raft_consensus.cc:695] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 4 LEADER]: Becoming Leader. State: Replica: e59304d21a1e44febcc404d724fc1577, State: Running, Role: LEADER
I20250831 07:57:56.949362 31538 consensus_queue.cc:237] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } }
I20250831 07:57:56.957819 31116 catalog_manager.cc:5582] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 reported cstate change: term changed from 3 to 4, leader changed from 2961283b97774862959c98dba71d8d78 (127.29.217.1) to e59304d21a1e44febcc404d724fc1577 (127.29.217.3). New cstate: current_term: 4 leader_uuid: "e59304d21a1e44febcc404d724fc1577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 } health_report { overall_health: HEALTHY } } }
I20250831 07:57:57.224081 31369 raft_consensus.cc:1273] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Refusing update from remote peer e175aaa0dcf3486984c542aaf3563df1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:57.225219 31535 consensus_queue.cc:1035] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:57:57.234247 31219 raft_consensus.cc:1273] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Refusing update from remote peer e175aaa0dcf3486984c542aaf3563df1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:57.235714 31535 consensus_queue.cc:1035] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:57:57.323683 31369 raft_consensus.cc:1273] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Refusing update from remote peer e175aaa0dcf3486984c542aaf3563df1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:57.325016 31535 consensus_queue.cc:1035] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59304d21a1e44febcc404d724fc1577" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:57:57.333599 31219 raft_consensus.cc:1273] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Refusing update from remote peer e175aaa0dcf3486984c542aaf3563df1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:57.334805 31535 consensus_queue.cc:1035] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:57:57.388260 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:57.394336 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:57.394697 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:57.396786 31294 raft_consensus.cc:1273] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Refusing update from remote peer e59304d21a1e44febcc404d724fc1577: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:57.397858 31585 consensus_queue.cc:1035] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e175aaa0dcf3486984c542aaf3563df1" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:57:57.408494 31219 raft_consensus.cc:1273] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Refusing update from remote peer e59304d21a1e44febcc404d724fc1577: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:57:57.410481 31538 consensus_queue.cc:1035] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2961283b97774862959c98dba71d8d78" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:57:58.395637 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:58.401120 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:58.401583 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:57:59.402410 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:57:59.408294 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:57:59.408742 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:58:00.409801 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:00.415019 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:58:00.415378 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:58:01.416225 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:01.422398 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:58:01.422884 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:58:02.423733 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:02.428992 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:58:02.429378 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:58:03.430173 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:03.436424 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:58:03.436792 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:58:04.437651 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:04.442723 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:58:04.443120 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:58:05.444068 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:05.449249 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 07:58:05.449635 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:58:06.477123 30564 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20250831 07:58:06.477556 30564 tablet_server.cc:178] TabletServer@127.29.217.1:0 shutting down...
I20250831 07:58:06.498835 30564 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250831 07:58:06.499498 30564 tablet_replica.cc:331] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78: stopping tablet replica
I20250831 07:58:06.500522 30564 raft_consensus.cc:2241] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 3 LEADER]: Raft consensus shutting down.
I20250831 07:58:06.501544 30564 raft_consensus.cc:2270] T e9ad3c047ea14d648689485283a199aa P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.503330 30564 tablet_replica.cc:331] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78: stopping tablet replica
I20250831 07:58:06.503808 30564 raft_consensus.cc:2241] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 4 LEADER]: Raft consensus shutting down.
I20250831 07:58:06.504539 30564 raft_consensus.cc:2270] T 3db868229f2e47d0af6034b101d57e7a P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.506258 30564 tablet_replica.cc:331] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78: stopping tablet replica
I20250831 07:58:06.506718 30564 raft_consensus.cc:2241] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.507099 30564 raft_consensus.cc:2270] T 32f97fdb66fd4c24bcdef839b4c3432a P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.508667 30564 tablet_replica.cc:331] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78: stopping tablet replica
I20250831 07:58:06.509131 30564 raft_consensus.cc:2241] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Raft consensus shutting down.
I20250831 07:58:06.509799 30564 raft_consensus.cc:2270] T d9002d7d65a14e49ad6c842c00ced7f8 P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.511312 30564 tablet_replica.cc:331] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78: stopping tablet replica
I20250831 07:58:06.511759 30564 raft_consensus.cc:2241] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.512142 30564 raft_consensus.cc:2270] T acc1b872a4834f68a4494a11d4487df5 P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.513597 30564 tablet_replica.cc:331] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78: stopping tablet replica
I20250831 07:58:06.514063 30564 raft_consensus.cc:2241] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 2 LEADER]: Raft consensus shutting down.
I20250831 07:58:06.514699 30564 raft_consensus.cc:2270] T e2415c634a154f269bf1f47ba35a97c3 P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.516480 30564 tablet_replica.cc:331] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78: stopping tablet replica
I20250831 07:58:06.516985 30564 raft_consensus.cc:2241] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.517457 30564 raft_consensus.cc:2270] T c6698eb6af2f4c888796b8e9f1003baf P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.519129 30564 tablet_replica.cc:331] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78: stopping tablet replica
I20250831 07:58:06.519578 30564 raft_consensus.cc:2241] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.519977 30564 raft_consensus.cc:2270] T 0d2c02218be24f76ac0c207781177c0c P 2961283b97774862959c98dba71d8d78 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.521765 30564 tablet_replica.cc:331] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78: stopping tablet replica
W20250831 07:58:06.521641 31334 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
I20250831 07:58:06.522370 30564 raft_consensus.cc:2241] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.522841 30564 raft_consensus.cc:2270] T b46e28f6af7247e182075273233a24bc P 2961283b97774862959c98dba71d8d78 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.524600 30564 tablet_replica.cc:331] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78: stopping tablet replica
W20250831 07:58:06.524658 31334 consensus_peers.cc:489] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 -> Peer 2961283b97774862959c98dba71d8d78 (127.29.217.1:43687): Couldn't send request to peer 2961283b97774862959c98dba71d8d78. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:06.525081 30564 raft_consensus.cc:2241] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.525542 30564 raft_consensus.cc:2270] T 7800f3d155894012ab8b8a3067479d5f P 2961283b97774862959c98dba71d8d78 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.553658 30564 tablet_server.cc:195] TabletServer@127.29.217.1:0 shutdown complete.
I20250831 07:58:06.573616 30564 tablet_server.cc:178] TabletServer@127.29.217.2:0 shutting down...
W20250831 07:58:06.589401 31334 consensus_peers.cc:489] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 -> Peer e175aaa0dcf3486984c542aaf3563df1 (127.29.217.2:44001): Couldn't send request to peer e175aaa0dcf3486984c542aaf3563df1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:06.598582 30564 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250831 07:58:06.599265 30564 tablet_replica.cc:331] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1: stopping tablet replica
I20250831 07:58:06.600242 30564 raft_consensus.cc:2241] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.600853 30564 raft_consensus.cc:2270] T d9002d7d65a14e49ad6c842c00ced7f8 P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.602535 30564 tablet_replica.cc:331] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1: stopping tablet replica
I20250831 07:58:06.603045 30564 raft_consensus.cc:2241] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.603425 30564 raft_consensus.cc:2270] T 32f97fdb66fd4c24bcdef839b4c3432a P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.605144 30564 tablet_replica.cc:331] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1: stopping tablet replica
I20250831 07:58:06.605647 30564 raft_consensus.cc:2241] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 2 LEADER]: Raft consensus shutting down.
I20250831 07:58:06.606362 30564 raft_consensus.cc:2270] T b46e28f6af7247e182075273233a24bc P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.608100 30564 tablet_replica.cc:331] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1: stopping tablet replica
I20250831 07:58:06.608543 30564 raft_consensus.cc:2241] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.608911 30564 raft_consensus.cc:2270] T 7800f3d155894012ab8b8a3067479d5f P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.610368 30564 tablet_replica.cc:331] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1: stopping tablet replica
I20250831 07:58:06.610814 30564 raft_consensus.cc:2241] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 4 LEADER]: Raft consensus shutting down.
I20250831 07:58:06.611451 30564 raft_consensus.cc:2270] T c6698eb6af2f4c888796b8e9f1003baf P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.613165 30564 tablet_replica.cc:331] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1: stopping tablet replica
I20250831 07:58:06.613602 30564 raft_consensus.cc:2241] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 4 LEADER]: Raft consensus shutting down.
I20250831 07:58:06.614228 30564 raft_consensus.cc:2270] T 0d2c02218be24f76ac0c207781177c0c P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.615715 30564 tablet_replica.cc:331] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1: stopping tablet replica
I20250831 07:58:06.616231 30564 raft_consensus.cc:2241] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.616616 30564 raft_consensus.cc:2270] T e9ad3c047ea14d648689485283a199aa P e175aaa0dcf3486984c542aaf3563df1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.618165 30564 tablet_replica.cc:331] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1: stopping tablet replica
I20250831 07:58:06.618645 30564 raft_consensus.cc:2241] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.619033 30564 raft_consensus.cc:2270] T acc1b872a4834f68a4494a11d4487df5 P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.620591 30564 tablet_replica.cc:331] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1: stopping tablet replica
I20250831 07:58:06.621023 30564 raft_consensus.cc:2241] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.621387 30564 raft_consensus.cc:2270] T e2415c634a154f269bf1f47ba35a97c3 P e175aaa0dcf3486984c542aaf3563df1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.622906 30564 tablet_replica.cc:331] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1: stopping tablet replica
I20250831 07:58:06.623312 30564 raft_consensus.cc:2241] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.623698 30564 raft_consensus.cc:2270] T 3db868229f2e47d0af6034b101d57e7a P e175aaa0dcf3486984c542aaf3563df1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.651023 30564 tablet_server.cc:195] TabletServer@127.29.217.2:0 shutdown complete.
I20250831 07:58:06.669211 30564 tablet_server.cc:178] TabletServer@127.29.217.3:0 shutting down...
I20250831 07:58:06.695988 30564 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250831 07:58:06.696640 30564 tablet_replica.cc:331] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577: stopping tablet replica
I20250831 07:58:06.697232 30564 raft_consensus.cc:2241] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [term 2 LEADER]: Raft consensus shutting down.
I20250831 07:58:06.698006 30564 raft_consensus.cc:2270] T 32f97fdb66fd4c24bcdef839b4c3432a P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.699627 30564 tablet_replica.cc:331] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577: stopping tablet replica
I20250831 07:58:06.700129 30564 raft_consensus.cc:2241] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.700500 30564 raft_consensus.cc:2270] T 3db868229f2e47d0af6034b101d57e7a P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.702030 30564 tablet_replica.cc:331] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577: stopping tablet replica
I20250831 07:58:06.702490 30564 raft_consensus.cc:2241] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.702880 30564 raft_consensus.cc:2270] T d9002d7d65a14e49ad6c842c00ced7f8 P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.704417 30564 tablet_replica.cc:331] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577: stopping tablet replica
I20250831 07:58:06.704872 30564 raft_consensus.cc:2241] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 3 LEADER]: Raft consensus shutting down.
I20250831 07:58:06.705538 30564 raft_consensus.cc:2270] T 7800f3d155894012ab8b8a3067479d5f P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.707080 30564 tablet_replica.cc:331] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577: stopping tablet replica
I20250831 07:58:06.707527 30564 raft_consensus.cc:2241] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.707978 30564 raft_consensus.cc:2270] T e9ad3c047ea14d648689485283a199aa P e59304d21a1e44febcc404d724fc1577 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.709420 30564 tablet_replica.cc:331] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577: stopping tablet replica
I20250831 07:58:06.709923 30564 raft_consensus.cc:2241] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.710284 30564 raft_consensus.cc:2270] T b46e28f6af7247e182075273233a24bc P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.712111 30564 tablet_replica.cc:331] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577: stopping tablet replica
I20250831 07:58:06.712678 30564 raft_consensus.cc:2241] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.713057 30564 raft_consensus.cc:2270] T c6698eb6af2f4c888796b8e9f1003baf P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.714507 30564 tablet_replica.cc:331] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577: stopping tablet replica
I20250831 07:58:06.714931 30564 raft_consensus.cc:2241] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.715310 30564 raft_consensus.cc:2270] T 0d2c02218be24f76ac0c207781177c0c P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.716781 30564 tablet_replica.cc:331] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577: stopping tablet replica
I20250831 07:58:06.717204 30564 raft_consensus.cc:2241] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 4 LEADER]: Raft consensus shutting down.
I20250831 07:58:06.717861 30564 raft_consensus.cc:2270] T acc1b872a4834f68a4494a11d4487df5 P e59304d21a1e44febcc404d724fc1577 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.719367 30564 tablet_replica.cc:331] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577: stopping tablet replica
I20250831 07:58:06.719791 30564 raft_consensus.cc:2241] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:06.720175 30564 raft_consensus.cc:2270] T e2415c634a154f269bf1f47ba35a97c3 P e59304d21a1e44febcc404d724fc1577 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.747963 30564 tablet_server.cc:195] TabletServer@127.29.217.3:0 shutdown complete.
I20250831 07:58:06.767562 30564 master.cc:561] Master@127.29.217.62:34445 shutting down...
I20250831 07:58:06.787653 30564 raft_consensus.cc:2241] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [term 1 LEADER]: Raft consensus shutting down.
I20250831 07:58:06.788345 30564 raft_consensus.cc:2270] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:06.788676 30564 tablet_replica.cc:331] T 00000000000000000000000000000000 P c3e825111c04425db0218a70b3653bf0: stopping tablet replica
I20250831 07:58:06.808086 30564 master.cc:583] Master@127.29.217.62:34445 shutdown complete.
2961283b97774862959c98dba71d8d78 5
e175aaa0dcf3486984c542aaf3563df1 4
e59304d21a1e44febcc404d724fc1577 1
2961283b97774862959c98dba71d8d78 3
e175aaa0dcf3486984c542aaf3563df1 4
e59304d21a1e44febcc404d724fc1577 3
2961283b97774862959c98dba71d8d78 8
e175aaa0dcf3486984c542aaf3563df1 1
e59304d21a1e44febcc404d724fc1577 1
2961283b97774862959c98dba71d8d78 4
e175aaa0dcf3486984c542aaf3563df1 3
e59304d21a1e44febcc404d724fc1577 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28806 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20250831 07:58:06.847594 30564 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.217.62:40065
I20250831 07:58:06.848774 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 07:58:06.854275 31606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:58:06.854298 31607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:58:06.855158 31609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:58:06.856163 30564 server_base.cc:1047] running on GCE node
I20250831 07:58:06.857048 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 07:58:06.857234 30564 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 07:58:06.857344 30564 hybrid_clock.cc:648] HybridClock initialized: now 1756627086857334 us; error 0 us; skew 500 ppm
I20250831 07:58:06.857779 30564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 07:58:06.860100 30564 webserver.cc:480] Webserver started at http://127.29.217.62:45583/ using document root <none> and password file <none>
I20250831 07:58:06.860518 30564 fs_manager.cc:362] Metadata directory not provided
I20250831 07:58:06.860669 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 07:58:06.860874 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 07:58:06.861943 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/master-0-root/instance:
uuid: "7a9e73e8611540ee91fdee861c75c975"
format_stamp: "Formatted at 2025-08-31 07:58:06 on dist-test-slave-87tg"
I20250831 07:58:06.866279 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20250831 07:58:06.869387 31614 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 07:58:06.870182 30564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250831 07:58:06.870409 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/master-0-root
uuid: "7a9e73e8611540ee91fdee861c75c975"
format_stamp: "Formatted at 2025-08-31 07:58:06 on dist-test-slave-87tg"
I20250831 07:58:06.870641 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 07:58:06.893489 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 07:58:06.894620 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 07:58:06.942271 30564 rpc_server.cc:307] RPC server started. Bound to: 127.29.217.62:40065
I20250831 07:58:06.942358 31666 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.217.62:40065 every 8 connection(s)
I20250831 07:58:06.947007 31667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:06.957617 31667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975: Bootstrap starting.
I20250831 07:58:06.962765 31667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:06.967569 31667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975: No bootstrap required, opened a new log
I20250831 07:58:06.969836 31667 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9e73e8611540ee91fdee861c75c975" member_type: VOTER }
I20250831 07:58:06.970338 31667 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:06.970584 31667 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9e73e8611540ee91fdee861c75c975, State: Initialized, Role: FOLLOWER
I20250831 07:58:06.971211 31667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9e73e8611540ee91fdee861c75c975" member_type: VOTER }
I20250831 07:58:06.971673 31667 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250831 07:58:06.971952 31667 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250831 07:58:06.972239 31667 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:06.976595 31667 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9e73e8611540ee91fdee861c75c975" member_type: VOTER }
I20250831 07:58:06.977123 31667 leader_election.cc:304] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a9e73e8611540ee91fdee861c75c975; no voters:
I20250831 07:58:06.978201 31667 leader_election.cc:290] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250831 07:58:06.978575 31670 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:06.979842 31670 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [term 1 LEADER]: Becoming Leader. State: Replica: 7a9e73e8611540ee91fdee861c75c975, State: Running, Role: LEADER
I20250831 07:58:06.980556 31670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9e73e8611540ee91fdee861c75c975" member_type: VOTER }
I20250831 07:58:06.981215 31667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 07:58:06.983659 31672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a9e73e8611540ee91fdee861c75c975. Latest consensus state: current_term: 1 leader_uuid: "7a9e73e8611540ee91fdee861c75c975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9e73e8611540ee91fdee861c75c975" member_type: VOTER } }
I20250831 07:58:06.983630 31671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a9e73e8611540ee91fdee861c75c975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9e73e8611540ee91fdee861c75c975" member_type: VOTER } }
I20250831 07:58:06.984336 31672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [sys.catalog]: This master's current role is: LEADER
I20250831 07:58:06.984484 31671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975 [sys.catalog]: This master's current role is: LEADER
I20250831 07:58:06.987043 31675 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250831 07:58:06.993173 31675 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250831 07:58:06.995105 30564 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250831 07:58:07.001924 31675 catalog_manager.cc:1349] Generated new cluster ID: 0b19a3254df440a4b9eafe044d080e5d
I20250831 07:58:07.002264 31675 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250831 07:58:07.025846 31675 catalog_manager.cc:1372] Generated new certificate authority record
I20250831 07:58:07.027091 31675 catalog_manager.cc:1506] Loading token signing keys...
I20250831 07:58:07.042909 31675 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975: Generated new TSK 0
I20250831 07:58:07.043529 31675 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250831 07:58:07.062007 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 07:58:07.069280 31688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:58:07.070781 31689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:58:07.071301 31691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:58:07.071580 30564 server_base.cc:1047] running on GCE node
I20250831 07:58:07.072690 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 07:58:07.072883 30564 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 07:58:07.073037 30564 hybrid_clock.cc:648] HybridClock initialized: now 1756627087073017 us; error 0 us; skew 500 ppm
I20250831 07:58:07.073542 30564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 07:58:07.075901 30564 webserver.cc:480] Webserver started at http://127.29.217.1:35213/ using document root <none> and password file <none>
I20250831 07:58:07.076372 30564 fs_manager.cc:362] Metadata directory not provided
I20250831 07:58:07.076570 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 07:58:07.076849 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 07:58:07.078126 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-0-root/instance:
uuid: "a0d845d01c8c4894895f66e3beae298c"
format_stamp: "Formatted at 2025-08-31 07:58:07 on dist-test-slave-87tg"
I20250831 07:58:07.082598 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20250831 07:58:07.085841 31696 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 07:58:07.086562 30564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250831 07:58:07.086853 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-0-root
uuid: "a0d845d01c8c4894895f66e3beae298c"
format_stamp: "Formatted at 2025-08-31 07:58:07 on dist-test-slave-87tg"
I20250831 07:58:07.087116 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 07:58:07.100921 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 07:58:07.102586 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 07:58:07.104442 30564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 07:58:07.107375 30564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 07:58:07.107627 30564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 07:58:07.107928 30564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 07:58:07.108103 30564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 07:58:07.146473 30564 rpc_server.cc:307] RPC server started. Bound to: 127.29.217.1:45027
I20250831 07:58:07.146566 31758 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.217.1:45027 every 8 connection(s)
I20250831 07:58:07.151165 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 07:58:07.159116 31763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:58:07.159612 31764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:58:07.162345 30564 server_base.cc:1047] running on GCE node
W20250831 07:58:07.162365 31766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:58:07.163995 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 07:58:07.164265 30564 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 07:58:07.164466 30564 hybrid_clock.cc:648] HybridClock initialized: now 1756627087164449 us; error 0 us; skew 500 ppm
I20250831 07:58:07.165114 30564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 07:58:07.167670 31759 heartbeater.cc:344] Connected to a master server at 127.29.217.62:40065
I20250831 07:58:07.168116 31759 heartbeater.cc:461] Registering TS with master...
I20250831 07:58:07.168458 30564 webserver.cc:480] Webserver started at http://127.29.217.2:42485/ using document root <none> and password file <none>
I20250831 07:58:07.169008 31759 heartbeater.cc:507] Master 127.29.217.62:40065 requested a full tablet report, sending...
I20250831 07:58:07.169095 30564 fs_manager.cc:362] Metadata directory not provided
I20250831 07:58:07.169462 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 07:58:07.169806 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 07:58:07.171188 31631 ts_manager.cc:194] Registered new tserver with Master: a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:58:07.171198 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-1-root/instance:
uuid: "7512ba2ed3a94982a22a75f51a647e65"
format_stamp: "Formatted at 2025-08-31 07:58:07 on dist-test-slave-87tg"
I20250831 07:58:07.173069 31631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59664
I20250831 07:58:07.176501 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250831 07:58:07.179714 31771 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 07:58:07.180438 30564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250831 07:58:07.180698 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-1-root
uuid: "7512ba2ed3a94982a22a75f51a647e65"
format_stamp: "Formatted at 2025-08-31 07:58:07 on dist-test-slave-87tg"
I20250831 07:58:07.180964 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 07:58:07.190382 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 07:58:07.191531 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 07:58:07.192967 30564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 07:58:07.195034 30564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 07:58:07.195209 30564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 07:58:07.195446 30564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 07:58:07.195597 30564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 07:58:07.234144 30564 rpc_server.cc:307] RPC server started. Bound to: 127.29.217.2:37261
I20250831 07:58:07.234233 31833 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.217.2:37261 every 8 connection(s)
I20250831 07:58:07.238960 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 07:58:07.244874 31837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 07:58:07.246377 31838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:58:07.248427 30564 server_base.cc:1047] running on GCE node
W20250831 07:58:07.249980 31840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:58:07.251024 31834 heartbeater.cc:344] Connected to a master server at 127.29.217.62:40065
I20250831 07:58:07.251047 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 07:58:07.251469 30564 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 07:58:07.251470 31834 heartbeater.cc:461] Registering TS with master...
I20250831 07:58:07.251711 30564 hybrid_clock.cc:648] HybridClock initialized: now 1756627087251693 us; error 0 us; skew 500 ppm
I20250831 07:58:07.252363 30564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 07:58:07.252343 31834 heartbeater.cc:507] Master 127.29.217.62:40065 requested a full tablet report, sending...
I20250831 07:58:07.254331 31631 ts_manager.cc:194] Registered new tserver with Master: 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:07.255105 30564 webserver.cc:480] Webserver started at http://127.29.217.3:45507/ using document root <none> and password file <none>
I20250831 07:58:07.255585 30564 fs_manager.cc:362] Metadata directory not provided
I20250831 07:58:07.255636 31631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59666
I20250831 07:58:07.255777 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 07:58:07.256184 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 07:58:07.257310 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-2-root/instance:
uuid: "8247a38b550649aa9cd6e7668dcaec10"
format_stamp: "Formatted at 2025-08-31 07:58:07 on dist-test-slave-87tg"
I20250831 07:58:07.261605 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250831 07:58:07.264830 31845 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 07:58:07.265496 30564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250831 07:58:07.265785 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-2-root
uuid: "8247a38b550649aa9cd6e7668dcaec10"
format_stamp: "Formatted at 2025-08-31 07:58:07 on dist-test-slave-87tg"
I20250831 07:58:07.266062 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 07:58:07.283747 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 07:58:07.284976 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 07:58:07.286535 30564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 07:58:07.288779 30564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 07:58:07.288975 30564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 07:58:07.289254 30564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 07:58:07.289414 30564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 07:58:07.329959 30564 rpc_server.cc:307] RPC server started. Bound to: 127.29.217.3:42585
I20250831 07:58:07.330062 31907 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.217.3:42585 every 8 connection(s)
I20250831 07:58:07.345088 31908 heartbeater.cc:344] Connected to a master server at 127.29.217.62:40065
I20250831 07:58:07.345454 31908 heartbeater.cc:461] Registering TS with master...
I20250831 07:58:07.346181 31908 heartbeater.cc:507] Master 127.29.217.62:40065 requested a full tablet report, sending...
I20250831 07:58:07.348073 31631 ts_manager.cc:194] Registered new tserver with Master: 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:07.348153 30564 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01498245s
I20250831 07:58:07.349390 31631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59674
I20250831 07:58:07.350811 30564 test_util.cc:276] Using random seed: -1651811326
I20250831 07:58:07.375487 31631 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59680:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250831 07:58:07.381518 31631 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250831 07:58:07.574381 31799 tablet_service.cc:1468] Processing CreateTablet for tablet 9ea3cc82a6af45f0a86e4cc5c5a90335 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250831 07:58:07.575980 31873 tablet_service.cc:1468] Processing CreateTablet for tablet 9ea3cc82a6af45f0a86e4cc5c5a90335 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250831 07:58:07.577327 31873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ea3cc82a6af45f0a86e4cc5c5a90335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.576999 31792 tablet_service.cc:1468] Processing CreateTablet for tablet be7a7e4ea6e1431ab37fb092f87cebd3 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250831 07:58:07.577917 31867 tablet_service.cc:1468] Processing CreateTablet for tablet 0cbf2ae2189542948a3ffb6b88a4fa21 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250831 07:58:07.578143 31793 tablet_service.cc:1468] Processing CreateTablet for tablet 0cbf2ae2189542948a3ffb6b88a4fa21 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250831 07:58:07.579236 31867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cbf2ae2189542948a3ffb6b88a4fa21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.579355 31798 tablet_service.cc:1468] Processing CreateTablet for tablet 9d510d241da34d25a50f2d65d17041a0 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250831 07:58:07.580476 31794 tablet_service.cc:1468] Processing CreateTablet for tablet d3d650c422ee41af95705f3fd876ddb8 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250831 07:58:07.581621 31797 tablet_service.cc:1468] Processing CreateTablet for tablet 25d573ed55e048098c90bcbb00d12fdb (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250831 07:58:07.582767 31795 tablet_service.cc:1468] Processing CreateTablet for tablet 3054807582f54efba5d989f569ab0666 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250831 07:58:07.583994 31869 tablet_service.cc:1468] Processing CreateTablet for tablet 3054807582f54efba5d989f569ab0666 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250831 07:58:07.585386 31869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3054807582f54efba5d989f569ab0666. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.586726 31870 tablet_service.cc:1468] Processing CreateTablet for tablet 5eac48bce84e483582f863716d5c26f9 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250831 07:58:07.588042 31870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eac48bce84e483582f863716d5c26f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.589010 31868 tablet_service.cc:1468] Processing CreateTablet for tablet d3d650c422ee41af95705f3fd876ddb8 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250831 07:58:07.590162 31871 tablet_service.cc:1468] Processing CreateTablet for tablet 25d573ed55e048098c90bcbb00d12fdb (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250831 07:58:07.585022 31796 tablet_service.cc:1468] Processing CreateTablet for tablet 5eac48bce84e483582f863716d5c26f9 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250831 07:58:07.592689 31799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ea3cc82a6af45f0a86e4cc5c5a90335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.592425 31872 tablet_service.cc:1468] Processing CreateTablet for tablet 9d510d241da34d25a50f2d65d17041a0 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250831 07:58:07.593322 31794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3d650c422ee41af95705f3fd876ddb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.593952 31798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d510d241da34d25a50f2d65d17041a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.594573 31795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3054807582f54efba5d989f569ab0666. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.594655 31866 tablet_service.cc:1468] Processing CreateTablet for tablet be7a7e4ea6e1431ab37fb092f87cebd3 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250831 07:58:07.595229 31792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be7a7e4ea6e1431ab37fb092f87cebd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.596174 31793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cbf2ae2189542948a3ffb6b88a4fa21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.596876 31868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3d650c422ee41af95705f3fd876ddb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.600170 31872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d510d241da34d25a50f2d65d17041a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.602468 31866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be7a7e4ea6e1431ab37fb092f87cebd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.604138 31797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25d573ed55e048098c90bcbb00d12fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.608214 31796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eac48bce84e483582f863716d5c26f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.612195 31871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25d573ed55e048098c90bcbb00d12fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.626511 31791 tablet_service.cc:1468] Processing CreateTablet for tablet 719b7e1bd5624bb9a08892e34c5e1b47 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250831 07:58:07.629530 31865 tablet_service.cc:1468] Processing CreateTablet for tablet 719b7e1bd5624bb9a08892e34c5e1b47 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250831 07:58:07.630869 31865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719b7e1bd5624bb9a08892e34c5e1b47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.636418 31864 tablet_service.cc:1468] Processing CreateTablet for tablet 064c41a8e5dc42e8bdb35225c6aae643 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250831 07:58:07.636981 31724 tablet_service.cc:1468] Processing CreateTablet for tablet 9ea3cc82a6af45f0a86e4cc5c5a90335 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250831 07:58:07.637760 31864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 064c41a8e5dc42e8bdb35225c6aae643. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.638263 31724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ea3cc82a6af45f0a86e4cc5c5a90335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.640602 31791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719b7e1bd5624bb9a08892e34c5e1b47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.641021 31723 tablet_service.cc:1468] Processing CreateTablet for tablet 9d510d241da34d25a50f2d65d17041a0 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250831 07:58:07.642431 31723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d510d241da34d25a50f2d65d17041a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.644982 31722 tablet_service.cc:1468] Processing CreateTablet for tablet 25d573ed55e048098c90bcbb00d12fdb (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250831 07:58:07.646617 31722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25d573ed55e048098c90bcbb00d12fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.649042 31721 tablet_service.cc:1468] Processing CreateTablet for tablet 5eac48bce84e483582f863716d5c26f9 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250831 07:58:07.651727 31721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eac48bce84e483582f863716d5c26f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.654264 31873 tablet_service.cc:1468] Processing CreateTablet for tablet 6d77811606c643a4970bc6b8a5f39fdb (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250831 07:58:07.655490 31873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d77811606c643a4970bc6b8a5f39fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.656888 31719 tablet_service.cc:1468] Processing CreateTablet for tablet d3d650c422ee41af95705f3fd876ddb8 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250831 07:58:07.658167 31719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3d650c422ee41af95705f3fd876ddb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.673578 31790 tablet_service.cc:1468] Processing CreateTablet for tablet 064c41a8e5dc42e8bdb35225c6aae643 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250831 07:58:07.673776 31715 tablet_service.cc:1468] Processing CreateTablet for tablet 064c41a8e5dc42e8bdb35225c6aae643 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250831 07:58:07.675853 31790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 064c41a8e5dc42e8bdb35225c6aae643. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.676097 31715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 064c41a8e5dc42e8bdb35225c6aae643. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.677022 31716 tablet_service.cc:1468] Processing CreateTablet for tablet 719b7e1bd5624bb9a08892e34c5e1b47 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250831 07:58:07.678316 31716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719b7e1bd5624bb9a08892e34c5e1b47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.686513 31718 tablet_service.cc:1468] Processing CreateTablet for tablet 0cbf2ae2189542948a3ffb6b88a4fa21 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250831 07:58:07.688860 31717 tablet_service.cc:1468] Processing CreateTablet for tablet be7a7e4ea6e1431ab37fb092f87cebd3 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250831 07:58:07.690811 31720 tablet_service.cc:1468] Processing CreateTablet for tablet 3054807582f54efba5d989f569ab0666 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250831 07:58:07.695855 31718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cbf2ae2189542948a3ffb6b88a4fa21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.700253 31720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3054807582f54efba5d989f569ab0666. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.702826 31799 tablet_service.cc:1468] Processing CreateTablet for tablet 2acd7b3ac5f94f53ac1ef93264415686 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250831 07:58:07.688814 31796 tablet_service.cc:1468] Processing CreateTablet for tablet 6d77811606c643a4970bc6b8a5f39fdb (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250831 07:58:07.706820 31796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d77811606c643a4970bc6b8a5f39fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.708884 31717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be7a7e4ea6e1431ab37fb092f87cebd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.722921 31721 tablet_service.cc:1468] Processing CreateTablet for tablet 6d77811606c643a4970bc6b8a5f39fdb (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250831 07:58:07.724308 31721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d77811606c643a4970bc6b8a5f39fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.731791 31923 tablet_bootstrap.cc:492] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:07.753623 31723 tablet_service.cc:1468] Processing CreateTablet for tablet 2acd7b3ac5f94f53ac1ef93264415686 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250831 07:58:07.754830 31870 tablet_service.cc:1468] Processing CreateTablet for tablet 4d055e59fc97471a93e491ee81afa361 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250831 07:58:07.755416 31723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2acd7b3ac5f94f53ac1ef93264415686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.755739 31799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2acd7b3ac5f94f53ac1ef93264415686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.758389 31790 tablet_service.cc:1468] Processing CreateTablet for tablet 660e7b2f56414f4fbd04a1929c99b390 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250831 07:58:07.728209 31872 tablet_service.cc:1468] Processing CreateTablet for tablet 2acd7b3ac5f94f53ac1ef93264415686 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250831 07:58:07.759366 31869 tablet_service.cc:1468] Processing CreateTablet for tablet 660e7b2f56414f4fbd04a1929c99b390 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250831 07:58:07.763809 31724 tablet_service.cc:1468] Processing CreateTablet for tablet 660e7b2f56414f4fbd04a1929c99b390 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250831 07:58:07.765935 31790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 660e7b2f56414f4fbd04a1929c99b390. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.764107 31872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2acd7b3ac5f94f53ac1ef93264415686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.766696 31925 tablet_bootstrap.cc:492] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:07.766780 31793 tablet_service.cc:1468] Processing CreateTablet for tablet fc393e9ddc4c42dfa208cc7251b0b15b (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250831 07:58:07.770668 31923 tablet_bootstrap.cc:654] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:07.767535 31869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 660e7b2f56414f4fbd04a1929c99b390. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.779565 31870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d055e59fc97471a93e491ee81afa361. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.780862 31716 tablet_service.cc:1468] Processing CreateTablet for tablet fc393e9ddc4c42dfa208cc7251b0b15b (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250831 07:58:07.774762 31724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 660e7b2f56414f4fbd04a1929c99b390. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.775250 31925 tablet_bootstrap.cc:654] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:07.797909 31793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc393e9ddc4c42dfa208cc7251b0b15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.759112 31715 tablet_service.cc:1468] Processing CreateTablet for tablet 4d055e59fc97471a93e491ee81afa361 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250831 07:58:07.816148 31797 tablet_service.cc:1468] Processing CreateTablet for tablet f2f76351ade3435092b6fac201772424 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250831 07:58:07.817615 31797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2f76351ade3435092b6fac201772424. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.817760 31869 tablet_service.cc:1468] Processing CreateTablet for tablet f2f76351ade3435092b6fac201772424 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250831 07:58:07.819213 31869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2f76351ade3435092b6fac201772424. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.814356 31792 tablet_service.cc:1468] Processing CreateTablet for tablet 4d055e59fc97471a93e491ee81afa361 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250831 07:58:07.819886 31790 tablet_service.cc:1468] Processing CreateTablet for tablet ca7b8b0d72c94ed386b217b936955022 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250831 07:58:07.816286 31870 tablet_service.cc:1468] Processing CreateTablet for tablet 9a40df3e72bd462091ca307d230a90ed (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250831 07:58:07.817505 31924 tablet_bootstrap.cc:492] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:07.826301 31923 tablet_bootstrap.cc:492] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:07.826777 31923 ts_tablet_manager.cc:1397] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.012s
I20250831 07:58:07.828804 31795 tablet_service.cc:1468] Processing CreateTablet for tablet 9a40df3e72bd462091ca307d230a90ed (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250831 07:58:07.829587 31923 raft_consensus.cc:357] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:07.830300 31923 raft_consensus.cc:383] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:07.830578 31923 raft_consensus.cc:738] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:07.831387 31923 consensus_queue.cc:260] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:07.834002 31719 tablet_service.cc:1468] Processing CreateTablet for tablet 9a40df3e72bd462091ca307d230a90ed (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250831 07:58:07.834975 31792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d055e59fc97471a93e491ee81afa361. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.835675 31870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a40df3e72bd462091ca307d230a90ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.836472 31865 tablet_service.cc:1468] Processing CreateTablet for tablet fc393e9ddc4c42dfa208cc7251b0b15b (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250831 07:58:07.837711 31865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc393e9ddc4c42dfa208cc7251b0b15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.841622 31790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca7b8b0d72c94ed386b217b936955022. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.810456 31716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc393e9ddc4c42dfa208cc7251b0b15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.849411 31720 tablet_service.cc:1468] Processing CreateTablet for tablet f2f76351ade3435092b6fac201772424 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250831 07:58:07.849854 31925 tablet_bootstrap.cc:492] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:07.850515 31925 ts_tablet_manager.cc:1397] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.084s user 0.000s sys 0.015s
I20250831 07:58:07.853294 31925 raft_consensus.cc:357] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:07.853987 31925 raft_consensus.cc:383] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:07.854272 31925 raft_consensus.cc:738] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:07.855072 31925 consensus_queue.cc:260] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:07.856122 31795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a40df3e72bd462091ca307d230a90ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.864074 31873 tablet_service.cc:1468] Processing CreateTablet for tablet ca7b8b0d72c94ed386b217b936955022 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250831 07:58:07.865394 31873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca7b8b0d72c94ed386b217b936955022. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.866041 31724 tablet_service.cc:1468] Processing CreateTablet for tablet ca7b8b0d72c94ed386b217b936955022 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250831 07:58:07.870662 31715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d055e59fc97471a93e491ee81afa361. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.871151 31796 tablet_service.cc:1468] Processing CreateTablet for tablet a187e6c5ed8f4214b4486aab25955909 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250831 07:58:07.871654 31924 tablet_bootstrap.cc:654] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:07.872454 31796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a187e6c5ed8f4214b4486aab25955909. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.876906 31867 tablet_service.cc:1468] Processing CreateTablet for tablet a187e6c5ed8f4214b4486aab25955909 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250831 07:58:07.878326 31867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a187e6c5ed8f4214b4486aab25955909. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.879139 31798 tablet_service.cc:1468] Processing CreateTablet for tablet 9e12d332258d4d0b8d0421b49ed5d0c0 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250831 07:58:07.880045 31719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a40df3e72bd462091ca307d230a90ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.880496 31798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e12d332258d4d0b8d0421b49ed5d0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.884024 31720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2f76351ade3435092b6fac201772424. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.947435 31724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca7b8b0d72c94ed386b217b936955022. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.889220 31834 heartbeater.cc:499] Master 127.29.217.62:40065 was elected leader, sending a full tablet report...
I20250831 07:58:07.891808 31791 tablet_service.cc:1468] Processing CreateTablet for tablet 9949d6ea1bea439fa2ca1fff1c5641bd (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250831 07:58:07.948621 31791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9949d6ea1bea439fa2ca1fff1c5641bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.891806 31866 tablet_service.cc:1468] Processing CreateTablet for tablet 9e12d332258d4d0b8d0421b49ed5d0c0 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250831 07:58:07.951375 31865 tablet_service.cc:1468] Processing CreateTablet for tablet fcbe3794f3fc40819ffbbfef02a071c3 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250831 07:58:07.952605 31866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e12d332258d4d0b8d0421b49ed5d0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.954294 31870 tablet_service.cc:1468] Processing CreateTablet for tablet 14ab15c60aaf42cf911851425eaab94d (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250831 07:58:07.892685 31868 tablet_service.cc:1468] Processing CreateTablet for tablet 9949d6ea1bea439fa2ca1fff1c5641bd (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250831 07:58:07.893625 31794 tablet_service.cc:1468] Processing CreateTablet for tablet 838ecfd85da44202b231fff2105cde0b (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250831 07:58:07.956604 31795 tablet_service.cc:1468] Processing CreateTablet for tablet fcbe3794f3fc40819ffbbfef02a071c3 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250831 07:58:07.957015 31794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 838ecfd85da44202b231fff2105cde0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.897557 31864 tablet_service.cc:1468] Processing CreateTablet for tablet 54f9ace15ae4422682ce241dda87dedf (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250831 07:58:07.897557 31871 tablet_service.cc:1468] Processing CreateTablet for tablet 838ecfd85da44202b231fff2105cde0b (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250831 07:58:07.913947 31721 tablet_service.cc:1468] Processing CreateTablet for tablet 9949d6ea1bea439fa2ca1fff1c5641bd (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250831 07:58:07.959110 31721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9949d6ea1bea439fa2ca1fff1c5641bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.921018 31923 ts_tablet_manager.cc:1428] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.094s user 0.011s sys 0.003s
I20250831 07:58:07.924927 31869 tablet_service.cc:1468] Processing CreateTablet for tablet f611ea44f17c403e818574844e13b946 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250831 07:58:07.928900 31908 heartbeater.cc:499] Master 127.29.217.62:40065 was elected leader, sending a full tablet report...
I20250831 07:58:07.938071 31717 tablet_service.cc:1468] Processing CreateTablet for tablet 838ecfd85da44202b231fff2105cde0b (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250831 07:58:07.937291 31872 tablet_service.cc:1468] Processing CreateTablet for tablet e93b827594384bcf80c7487b061b4c24 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250831 07:58:07.941124 31793 tablet_service.cc:1468] Processing CreateTablet for tablet 54f9ace15ae4422682ce241dda87dedf (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250831 07:58:07.941270 31799 tablet_service.cc:1468] Processing CreateTablet for tablet f611ea44f17c403e818574844e13b946 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250831 07:58:07.972774 31717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 838ecfd85da44202b231fff2105cde0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.944700 31718 tablet_service.cc:1468] Processing CreateTablet for tablet 9e12d332258d4d0b8d0421b49ed5d0c0 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250831 07:58:07.973691 31718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e12d332258d4d0b8d0421b49ed5d0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.982955 31928 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:07.983477 31928 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:07.988179 31928 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:07.995395 31680 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 07:58:07.945896 31797 tablet_service.cc:1468] Processing CreateTablet for tablet e93b827594384bcf80c7487b061b4c24 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250831 07:58:07.946666 31925 ts_tablet_manager.cc:1428] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.096s user 0.011s sys 0.000s
I20250831 07:58:07.998438 31925 tablet_bootstrap.cc:492] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:07.998260 31723 tablet_service.cc:1468] Processing CreateTablet for tablet 54f9ace15ae4422682ce241dda87dedf (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250831 07:58:07.957773 31795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcbe3794f3fc40819ffbbfef02a071c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.002318 31723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54f9ace15ae4422682ce241dda87dedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:07.884162 31722 tablet_service.cc:1468] Processing CreateTablet for tablet a187e6c5ed8f4214b4486aab25955909 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250831 07:58:08.053841 31722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a187e6c5ed8f4214b4486aab25955909. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.060585 31790 tablet_service.cc:1468] Processing CreateTablet for tablet 14ab15c60aaf42cf911851425eaab94d (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250831 07:58:08.067382 31716 tablet_service.cc:1468] Processing CreateTablet for tablet f611ea44f17c403e818574844e13b946 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250831 07:58:08.068691 31719 tablet_service.cc:1468] Processing CreateTablet for tablet e93b827594384bcf80c7487b061b4c24 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250831 07:58:08.057430 31797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e93b827594384bcf80c7487b061b4c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.063117 31724 tablet_service.cc:1468] Processing CreateTablet for tablet 14ab15c60aaf42cf911851425eaab94d (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250831 07:58:08.068739 31716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f611ea44f17c403e818574844e13b946. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.072103 31790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14ab15c60aaf42cf911851425eaab94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.080765 31924 tablet_bootstrap.cc:492] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:08.081296 31924 ts_tablet_manager.cc:1397] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.264s user 0.025s sys 0.033s
I20250831 07:58:08.084174 31929 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:08.087263 31929 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.091415 31929 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:08.085992 31924 raft_consensus.cc:357] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.094672 31923 tablet_bootstrap.cc:492] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:08.104068 31867 tablet_service.cc:1468] Processing CreateTablet for tablet 855219eabb9f4417ad63ec7efa7982da (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250831 07:58:08.093261 31868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9949d6ea1bea439fa2ca1fff1c5641bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.106410 31798 tablet_service.cc:1468] Processing CreateTablet for tablet f7c92dde8fc9490dafb6d51ad8b26855 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250831 07:58:08.107755 31865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcbe3794f3fc40819ffbbfef02a071c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.110639 31717 tablet_service.cc:1468] Processing CreateTablet for tablet 855219eabb9f4417ad63ec7efa7982da (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250831 07:58:08.112147 31872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e93b827594384bcf80c7487b061b4c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.093636 31924 raft_consensus.cc:383] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:08.137115 31717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 855219eabb9f4417ad63ec7efa7982da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.139163 31869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f611ea44f17c403e818574844e13b946. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.140151 31793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54f9ace15ae4422682ce241dda87dedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.092631 31796 tablet_service.cc:1468] Processing CreateTablet for tablet 90effb264f51485dac220ee32f49e4ad (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250831 07:58:08.139238 31924 raft_consensus.cc:738] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:08.153268 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:08.155634 31849 leader_election.cc:343] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.156109 31867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 855219eabb9f4417ad63ec7efa7982da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.157378 31721 tablet_service.cc:1468] Processing CreateTablet for tablet 90effb264f51485dac220ee32f49e4ad (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250831 07:58:08.158951 31866 tablet_service.cc:1468] Processing CreateTablet for tablet 90effb264f51485dac220ee32f49e4ad (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250831 07:58:08.160106 31871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 838ecfd85da44202b231fff2105cde0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.161983 31720 tablet_service.cc:1468] Processing CreateTablet for tablet f7c92dde8fc9490dafb6d51ad8b26855 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250831 07:58:08.096002 31792 tablet_service.cc:1468] Processing CreateTablet for tablet 855219eabb9f4417ad63ec7efa7982da (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250831 07:58:08.164120 31864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54f9ace15ae4422682ce241dda87dedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.165117 31924 consensus_queue.cc:260] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.168375 31718 tablet_service.cc:1468] Processing CreateTablet for tablet fcbe3794f3fc40819ffbbfef02a071c3 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250831 07:58:08.142845 31791 tablet_service.cc:1468] Processing CreateTablet for tablet 000f9ab653554f498fe14065e6656a62 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250831 07:58:08.171155 31873 tablet_service.cc:1468] Processing CreateTablet for tablet 000f9ab653554f498fe14065e6656a62 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250831 07:58:08.172895 31715 tablet_service.cc:1468] Processing CreateTablet for tablet 000f9ab653554f498fe14065e6656a62 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250831 07:58:08.175657 31759 heartbeater.cc:499] Master 127.29.217.62:40065 was elected leader, sending a full tablet report...
I20250831 07:58:08.177937 31724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14ab15c60aaf42cf911851425eaab94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.181912 31720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c92dde8fc9490dafb6d51ad8b26855. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.183974 31718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcbe3794f3fc40819ffbbfef02a071c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.190271 31925 tablet_bootstrap.cc:654] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:08.201951 31923 tablet_bootstrap.cc:654] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:08.203919 31799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f611ea44f17c403e818574844e13b946. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.204692 31798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c92dde8fc9490dafb6d51ad8b26855. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.207510 31796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90effb264f51485dac220ee32f49e4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.208315 31792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 855219eabb9f4417ad63ec7efa7982da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.208724 31719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e93b827594384bcf80c7487b061b4c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.210008 31721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90effb264f51485dac220ee32f49e4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.210783 31870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14ab15c60aaf42cf911851425eaab94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.212848 31715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 000f9ab653554f498fe14065e6656a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.216825 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:08.217132 31790 tablet_service.cc:1468] Processing CreateTablet for tablet 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250831 07:58:08.218364 31790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e9fcfb2ddfa4c6eb4e9264f55b5dba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.219698 31866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90effb264f51485dac220ee32f49e4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.220366 31791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 000f9ab653554f498fe14065e6656a62. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 07:58:08.224291 31775 leader_election.cc:343] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.233244 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:08.235469 31848 leader_election.cc:343] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.235926 31848 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:08.245718 31873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 000f9ab653554f498fe14065e6656a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.252408 31928 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:08.283761 31799 tablet_service.cc:1468] Processing CreateTablet for tablet 2068a5cd20ba438a9dae813ac5acda03 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250831 07:58:08.284237 31795 tablet_service.cc:1468] Processing CreateTablet for tablet ccd51d2e599e4535ad0f2f6afcb1bfee (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250831 07:58:08.285141 31799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2068a5cd20ba438a9dae813ac5acda03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.287911 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:08.292924 31871 tablet_service.cc:1468] Processing CreateTablet for tablet ccd51d2e599e4535ad0f2f6afcb1bfee (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250831 07:58:08.294193 31871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd51d2e599e4535ad0f2f6afcb1bfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.299088 31865 tablet_service.cc:1468] Processing CreateTablet for tablet 982febd23d7540aaa3b48b70133e982e (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250831 07:58:08.239461 31868 tablet_service.cc:1468] Processing CreateTablet for tablet f7c92dde8fc9490dafb6d51ad8b26855 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250831 07:58:08.306279 31795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd51d2e599e4535ad0f2f6afcb1bfee. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 07:58:08.312181 31773 leader_election.cc:343] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.312507 31799 tablet_service.cc:1468] Processing CreateTablet for tablet 982febd23d7540aaa3b48b70133e982e (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250831 07:58:08.312724 31773 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:08.314085 31929 raft_consensus.cc:2747] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:08.313982 31872 tablet_service.cc:1468] Processing CreateTablet for tablet 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250831 07:58:08.316355 31799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 982febd23d7540aaa3b48b70133e982e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.329526 31716 tablet_service.cc:1468] Processing CreateTablet for tablet 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250831 07:58:08.331283 31722 tablet_service.cc:1468] Processing CreateTablet for tablet ccd51d2e599e4535ad0f2f6afcb1bfee (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250831 07:58:08.336700 31792 tablet_service.cc:1468] Processing CreateTablet for tablet b606cb7b5ccc4e2588ca98715c021620 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250831 07:58:08.338023 31792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b606cb7b5ccc4e2588ca98715c021620. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.343485 31924 ts_tablet_manager.cc:1428] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.262s user 0.014s sys 0.008s
I20250831 07:58:08.343438 31723 tablet_service.cc:1468] Processing CreateTablet for tablet 2068a5cd20ba438a9dae813ac5acda03 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250831 07:58:08.347982 31869 tablet_service.cc:1468] Processing CreateTablet for tablet 2068a5cd20ba438a9dae813ac5acda03 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250831 07:58:08.356626 31797 tablet_service.cc:1468] Processing CreateTablet for tablet db51a15157664ec0a9d8402ca441f042 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250831 07:58:08.357964 31797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db51a15157664ec0a9d8402ca441f042. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.358084 31793 tablet_service.cc:1468] Processing CreateTablet for tablet c3bdbd8499374432a94ff0909a4c5f95 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250831 07:58:08.359318 31793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3bdbd8499374432a94ff0909a4c5f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.361984 31865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 982febd23d7540aaa3b48b70133e982e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.363171 31791 tablet_service.cc:1468] Processing CreateTablet for tablet 9e3be3f093d144268837fd81ceb8b44a (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250831 07:58:08.364199 31720 tablet_service.cc:1468] Processing CreateTablet for tablet 982febd23d7540aaa3b48b70133e982e (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250831 07:58:08.365965 31722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd51d2e599e4535ad0f2f6afcb1bfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.366744 31720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 982febd23d7540aaa3b48b70133e982e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.368530 31723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2068a5cd20ba438a9dae813ac5acda03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.368695 31924 tablet_bootstrap.cc:492] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:08.369782 31716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e9fcfb2ddfa4c6eb4e9264f55b5dba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.375674 31791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e3be3f093d144268837fd81ceb8b44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.377470 31925 tablet_bootstrap.cc:492] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:08.389623 31925 ts_tablet_manager.cc:1397] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.391s user 0.023s sys 0.028s
I20250831 07:58:08.389463 31794 tablet_service.cc:1468] Processing CreateTablet for tablet e3a3b559f26e43f8942b8753f3e30607 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250831 07:58:08.406431 31794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a3b559f26e43f8942b8753f3e30607. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.406754 31867 tablet_service.cc:1468] Processing CreateTablet for tablet b606cb7b5ccc4e2588ca98715c021620 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250831 07:58:08.408313 31867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b606cb7b5ccc4e2588ca98715c021620. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.413245 31869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2068a5cd20ba438a9dae813ac5acda03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.414227 31924 tablet_bootstrap.cc:654] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:08.414278 31717 tablet_service.cc:1468] Processing CreateTablet for tablet 9e3be3f093d144268837fd81ceb8b44a (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250831 07:58:08.416612 31798 tablet_service.cc:1468] Processing CreateTablet for tablet 518e6f5a91554de78e0bd0b6ec04c12a (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250831 07:58:08.418007 31798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 518e6f5a91554de78e0bd0b6ec04c12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.418529 31872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e9fcfb2ddfa4c6eb4e9264f55b5dba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.418004 31866 tablet_service.cc:1468] Processing CreateTablet for tablet db51a15157664ec0a9d8402ca441f042 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250831 07:58:08.421082 31866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db51a15157664ec0a9d8402ca441f042. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.421267 31870 tablet_service.cc:1468] Processing CreateTablet for tablet e3a3b559f26e43f8942b8753f3e30607 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250831 07:58:08.422400 31870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a3b559f26e43f8942b8753f3e30607. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.422909 31923 tablet_bootstrap.cc:492] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:08.423427 31923 ts_tablet_manager.cc:1397] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.329s user 0.019s sys 0.011s
I20250831 07:58:08.424440 31868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c92dde8fc9490dafb6d51ad8b26855. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.425215 31864 tablet_service.cc:1468] Processing CreateTablet for tablet 9e3be3f093d144268837fd81ceb8b44a (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250831 07:58:08.426540 31864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e3be3f093d144268837fd81ceb8b44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.389464 31718 tablet_service.cc:1468] Processing CreateTablet for tablet b606cb7b5ccc4e2588ca98715c021620 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250831 07:58:08.429255 31717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e3be3f093d144268837fd81ceb8b44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.430094 31718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b606cb7b5ccc4e2588ca98715c021620. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.448205 31923 raft_consensus.cc:357] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:08.450428 31923 raft_consensus.cc:383] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:08.450801 31923 raft_consensus.cc:738] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:08.392742 31925 raft_consensus.cc:357] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.397655 31715 tablet_service.cc:1468] Processing CreateTablet for tablet db51a15157664ec0a9d8402ca441f042 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250831 07:58:08.452014 31925 raft_consensus.cc:383] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:08.452337 31925 raft_consensus.cc:738] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:08.452735 31715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db51a15157664ec0a9d8402ca441f042. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.453030 31925 consensus_queue.cc:260] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.455294 31925 ts_tablet_manager.cc:1428] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.065s user 0.003s sys 0.003s
I20250831 07:58:08.456763 31925 tablet_bootstrap.cc:492] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:08.399432 31724 tablet_service.cc:1468] Processing CreateTablet for tablet e3a3b559f26e43f8942b8753f3e30607 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250831 07:58:08.459090 31724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a3b559f26e43f8942b8753f3e30607. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.459514 31873 tablet_service.cc:1468] Processing CreateTablet for tablet c3bdbd8499374432a94ff0909a4c5f95 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250831 07:58:08.460843 31873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3bdbd8499374432a94ff0909a4c5f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.463361 31925 tablet_bootstrap.cc:654] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:08.464954 31721 tablet_service.cc:1468] Processing CreateTablet for tablet 518e6f5a91554de78e0bd0b6ec04c12a (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250831 07:58:08.466259 31721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 518e6f5a91554de78e0bd0b6ec04c12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.466920 31720 tablet_service.cc:1468] Processing CreateTablet for tablet c3bdbd8499374432a94ff0909a4c5f95 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250831 07:58:08.475003 31720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3bdbd8499374432a94ff0909a4c5f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.466081 31923 consensus_queue.cc:260] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:08.482425 31923 ts_tablet_manager.cc:1428] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.059s user 0.006s sys 0.007s
I20250831 07:58:08.391433 31796 tablet_service.cc:1468] Processing CreateTablet for tablet a62a36682136474ca552fbc53190e09e (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250831 07:58:08.483500 31923 tablet_bootstrap.cc:492] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:08.483783 31796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a62a36682136474ca552fbc53190e09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.485555 31719 tablet_service.cc:1468] Processing CreateTablet for tablet a62a36682136474ca552fbc53190e09e (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250831 07:58:08.487010 31719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a62a36682136474ca552fbc53190e09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.490621 31791 tablet_service.cc:1468] Processing CreateTablet for tablet 9560a731dd294178911faa26dbfc2ef8 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250831 07:58:08.491967 31791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9560a731dd294178911faa26dbfc2ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.492749 31795 tablet_service.cc:1468] Processing CreateTablet for tablet 26b32ddce4444eb89ebd7a73a7900bdf (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250831 07:58:08.494015 31795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26b32ddce4444eb89ebd7a73a7900bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.498222 31923 tablet_bootstrap.cc:654] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:08.498428 31790 tablet_service.cc:1468] Processing CreateTablet for tablet bb2667d2ec934433ac10d395c1d29db4 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250831 07:58:08.510537 31790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb2667d2ec934433ac10d395c1d29db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.519044 31716 tablet_service.cc:1468] Processing CreateTablet for tablet 9560a731dd294178911faa26dbfc2ef8 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250831 07:58:08.490722 31871 tablet_service.cc:1468] Processing CreateTablet for tablet 518e6f5a91554de78e0bd0b6ec04c12a (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250831 07:58:08.521981 31871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 518e6f5a91554de78e0bd0b6ec04c12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.527648 31792 tablet_service.cc:1468] Processing CreateTablet for tablet 59e33bfa463944f89cfbb00d1247323a (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250831 07:58:08.528968 31792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e33bfa463944f89cfbb00d1247323a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.542465 31924 tablet_bootstrap.cc:492] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:08.543021 31924 ts_tablet_manager.cc:1397] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.175s user 0.006s sys 0.010s
I20250831 07:58:08.552559 31792 tablet_service.cc:1468] Processing CreateTablet for tablet 4f319de3ac3c4b9a9b87cd497e02c51c (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250831 07:58:08.553968 31792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f319de3ac3c4b9a9b87cd497e02c51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.558828 31717 tablet_service.cc:1468] Processing CreateTablet for tablet 26b32ddce4444eb89ebd7a73a7900bdf (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250831 07:58:08.561606 31717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26b32ddce4444eb89ebd7a73a7900bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.558974 31869 tablet_service.cc:1468] Processing CreateTablet for tablet 26b32ddce4444eb89ebd7a73a7900bdf (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250831 07:58:08.568635 31869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26b32ddce4444eb89ebd7a73a7900bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.572842 31799 tablet_service.cc:1468] Processing CreateTablet for tablet 9d3848dba46b4b239a92dc14f93ef48d (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250831 07:58:08.574189 31799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d3848dba46b4b239a92dc14f93ef48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.568266 31864 tablet_service.cc:1468] Processing CreateTablet for tablet 59e33bfa463944f89cfbb00d1247323a (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250831 07:58:08.584606 31722 tablet_service.cc:1468] Processing CreateTablet for tablet bb2667d2ec934433ac10d395c1d29db4 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250831 07:58:08.590628 31937 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:08.591143 31937 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.609225 31796 tablet_service.cc:1468] Processing CreateTablet for tablet 3f70249b7dd14c0e804b05303fa06f09 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250831 07:58:08.610598 31796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f70249b7dd14c0e804b05303fa06f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.610443 31865 tablet_service.cc:1468] Processing CreateTablet for tablet a62a36682136474ca552fbc53190e09e (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250831 07:58:08.596612 31937 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:08.602327 31716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9560a731dd294178911faa26dbfc2ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.633064 31790 tablet_service.cc:1468] Processing CreateTablet for tablet 2b32e93c114645eeb4e707778c91e12e (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250831 07:58:08.633898 31925 tablet_bootstrap.cc:492] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:08.634361 31925 ts_tablet_manager.cc:1397] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.178s user 0.017s sys 0.008s
I20250831 07:58:08.634460 31790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b32e93c114645eeb4e707778c91e12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.637914 31925 raft_consensus.cc:357] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:08.639186 31925 raft_consensus.cc:383] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:08.639595 31925 raft_consensus.cc:738] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:08.639791 31924 raft_consensus.cc:357] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.640501 31925 consensus_queue.cc:260] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:08.641048 31924 raft_consensus.cc:383] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:08.641489 31924 raft_consensus.cc:738] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:08.650700 31925 ts_tablet_manager.cc:1428] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20250831 07:58:08.650786 31867 tablet_service.cc:1468] Processing CreateTablet for tablet 9560a731dd294178911faa26dbfc2ef8 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250831 07:58:08.651933 31925 tablet_bootstrap.cc:492] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:08.642390 31924 consensus_queue.cc:260] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.653975 31924 ts_tablet_manager.cc:1428] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.111s user 0.010s sys 0.000s
I20250831 07:58:08.654943 31924 tablet_bootstrap.cc:492] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:08.656123 31722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb2667d2ec934433ac10d395c1d29db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.659180 31925 tablet_bootstrap.cc:654] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:08.660357 31720 tablet_service.cc:1468] Processing CreateTablet for tablet 59e33bfa463944f89cfbb00d1247323a (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250831 07:58:08.661626 31720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e33bfa463944f89cfbb00d1247323a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.664446 31798 tablet_service.cc:1468] Processing CreateTablet for tablet 95d9a42ca12741e382b05c0cd9007bfc (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250831 07:58:08.665709 31798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d9a42ca12741e382b05c0cd9007bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.669133 31715 tablet_service.cc:1468] Processing CreateTablet for tablet 95d9a42ca12741e382b05c0cd9007bfc (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250831 07:58:08.670343 31715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d9a42ca12741e382b05c0cd9007bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.683332 31718 tablet_service.cc:1468] Processing CreateTablet for tablet 9d3848dba46b4b239a92dc14f93ef48d (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250831 07:58:08.644274 31864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e33bfa463944f89cfbb00d1247323a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.684897 31794 tablet_service.cc:1468] Processing CreateTablet for tablet 6b2e336f43934f118ecacdc1e1d2da60 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250831 07:58:08.686895 31794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b2e336f43934f118ecacdc1e1d2da60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.702138 31867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9560a731dd294178911faa26dbfc2ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.704252 31718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d3848dba46b4b239a92dc14f93ef48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.710001 31793 tablet_service.cc:1468] Processing CreateTablet for tablet 409f8ff1108b412382695bc6ce64e3cc (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250831 07:58:08.711315 31793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 409f8ff1108b412382695bc6ce64e3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.715809 31792 tablet_service.cc:1468] Processing CreateTablet for tablet 953dd19ad1e4414ab1c956437fd8bbfa (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250831 07:58:08.715811 31795 tablet_service.cc:1468] Processing CreateTablet for tablet 26525016efb247c3b1c2bf7fec0e1658 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250831 07:58:08.718755 31792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 953dd19ad1e4414ab1c956437fd8bbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.724182 31923 tablet_bootstrap.cc:492] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:08.724692 31923 ts_tablet_manager.cc:1397] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.241s user 0.012s sys 0.022s
I20250831 07:58:08.728391 31717 tablet_service.cc:1468] Processing CreateTablet for tablet 2b32e93c114645eeb4e707778c91e12e (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250831 07:58:08.729689 31717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b32e93c114645eeb4e707778c91e12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.727463 31923 raft_consensus.cc:357] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.733601 31923 raft_consensus.cc:383] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:08.734108 31923 raft_consensus.cc:738] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:08.732918 31721 tablet_service.cc:1468] Processing CreateTablet for tablet 3f70249b7dd14c0e804b05303fa06f09 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250831 07:58:08.736119 31721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f70249b7dd14c0e804b05303fa06f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.737730 31719 tablet_service.cc:1468] Processing CreateTablet for tablet 6b2e336f43934f118ecacdc1e1d2da60 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250831 07:58:08.736666 31923 consensus_queue.cc:260] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.739114 31719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b2e336f43934f118ecacdc1e1d2da60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.739043 31928 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:08.739573 31928 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.740417 31795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26525016efb247c3b1c2bf7fec0e1658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.742835 31937 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:08.743403 31937 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.745353 31937 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:08.747005 31872 tablet_service.cc:1468] Processing CreateTablet for tablet bb2667d2ec934433ac10d395c1d29db4 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250831 07:58:08.748862 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:08.749284 31797 tablet_service.cc:1468] Processing CreateTablet for tablet 91390ee2f1e2445ca47e8cb7bd8eeecd (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
W20250831 07:58:08.750372 31849 leader_election.cc:343] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.750514 31797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91390ee2f1e2445ca47e8cb7bd8eeecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.755043 31724 tablet_service.cc:1468] Processing CreateTablet for tablet 409f8ff1108b412382695bc6ce64e3cc (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250831 07:58:08.756326 31724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 409f8ff1108b412382695bc6ce64e3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.757253 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:08.758602 31848 leader_election.cc:343] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.759023 31848 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:08.760069 31928 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:08.760660 31928 raft_consensus.cc:2747] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:08.718005 31865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a62a36682136474ca552fbc53190e09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.765543 31923 ts_tablet_manager.cc:1428] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20250831 07:58:08.775202 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:08.775216 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:08.778921 31699 leader_election.cc:343] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.779332 31928 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:08.779639 31699 leader_election.cc:343] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.779785 31928 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:08.784179 31928 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:08.785717 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:08.787084 31849 leader_election.cc:343] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.787351 31925 tablet_bootstrap.cc:492] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:08.787801 31925 ts_tablet_manager.cc:1397] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.136s user 0.010s sys 0.010s
I20250831 07:58:08.788072 31723 tablet_service.cc:1468] Processing CreateTablet for tablet 4f319de3ac3c4b9a9b87cd497e02c51c (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250831 07:58:08.790508 31925 raft_consensus.cc:357] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.791200 31925 raft_consensus.cc:383] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:08.791468 31925 raft_consensus.cc:738] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:08.792439 31929 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:08.792891 31929 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.794778 31929 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:08.795557 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:08.795739 31925 consensus_queue.cc:260] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:08.796941 31775 leader_election.cc:343] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.798627 31925 ts_tablet_manager.cc:1428] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250831 07:58:08.799676 31925 tablet_bootstrap.cc:492] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:08.804596 31723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f319de3ac3c4b9a9b87cd497e02c51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.805936 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:08.816248 31809 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:08.818543 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:08.818388 31870 tablet_service.cc:1468] Processing CreateTablet for tablet 9d3848dba46b4b239a92dc14f93ef48d (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250831 07:58:08.820315 31923 tablet_bootstrap.cc:492] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:08.820776 31929 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:08.821228 31929 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:08.823694 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:08.824251 31929 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:08.824980 31775 leader_election.cc:343] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.825150 31866 tablet_service.cc:1468] Processing CreateTablet for tablet 4f319de3ac3c4b9a9b87cd497e02c51c (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250831 07:58:08.826493 31716 tablet_service.cc:1468] Processing CreateTablet for tablet 26525016efb247c3b1c2bf7fec0e1658 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250831 07:58:08.827746 31716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26525016efb247c3b1c2bf7fec0e1658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.828078 31872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb2667d2ec934433ac10d395c1d29db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.833467 31924 tablet_bootstrap.cc:654] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:08.833119 31724 tablet_service.cc:1468] Processing CreateTablet for tablet 953dd19ad1e4414ab1c956437fd8bbfa (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250831 07:58:08.836570 31724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 953dd19ad1e4414ab1c956437fd8bbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.836946 31791 tablet_service.cc:1468] Processing CreateTablet for tablet 3eb1971f5bb14c27bdbad34649f50ab4 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250831 07:58:08.838209 31791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eb1971f5bb14c27bdbad34649f50ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.837918 31790 tablet_service.cc:1468] Processing CreateTablet for tablet a98f72c022fe465da362553182e4ecca (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250831 07:58:08.838774 31923 tablet_bootstrap.cc:654] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:08.839519 31790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98f72c022fe465da362553182e4ecca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.841148 31794 tablet_service.cc:1468] Processing CreateTablet for tablet fe2bab0189fe475cb540bdf7a041d8b8 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250831 07:58:08.842288 31794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe2bab0189fe475cb540bdf7a041d8b8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 07:58:08.842502 31773 leader_election.cc:343] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.842960 31773 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:08.845857 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:08.846942 31848 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:08.847700 31929 raft_consensus.cc:2747] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:08.851037 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:08.852614 31698 leader_election.cc:343] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.853065 31698 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:08.853825 31937 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:08.861766 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:08.861713 31868 tablet_service.cc:1468] Processing CreateTablet for tablet 95d9a42ca12741e382b05c0cd9007bfc (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
W20250831 07:58:08.863075 31698 leader_election.cc:343] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.863571 31698 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:08.864429 31937 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:08.863061 31868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d9a42ca12741e382b05c0cd9007bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.869447 31873 tablet_service.cc:1468] Processing CreateTablet for tablet 3f70249b7dd14c0e804b05303fa06f09 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250831 07:58:08.870880 31873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f70249b7dd14c0e804b05303fa06f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.872385 31870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d3848dba46b4b239a92dc14f93ef48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.876164 31946 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:08.876595 31946 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:08.876996 31946 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:08.880678 31773 leader_election.cc:343] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:08.881213 31773 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:08.882887 31929 raft_consensus.cc:2747] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:08.882110 31716 tablet_service.cc:1468] Processing CreateTablet for tablet 91390ee2f1e2445ca47e8cb7bd8eeecd (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250831 07:58:08.888867 31716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91390ee2f1e2445ca47e8cb7bd8eeecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.889338 31946 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:08.890794 31866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f319de3ac3c4b9a9b87cd497e02c51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.895112 31925 tablet_bootstrap.cc:654] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:08.906608 31717 tablet_service.cc:1468] Processing CreateTablet for tablet a98f72c022fe465da362553182e4ecca (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250831 07:58:08.907948 31717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98f72c022fe465da362553182e4ecca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.893165 31724 tablet_service.cc:1468] Processing CreateTablet for tablet 3eb1971f5bb14c27bdbad34649f50ab4 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250831 07:58:08.912705 31724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eb1971f5bb14c27bdbad34649f50ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:08.925802 31924 tablet_bootstrap.cc:492] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:08.926425 31924 ts_tablet_manager.cc:1397] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.272s user 0.008s sys 0.006s
I20250831 07:58:08.931466 31864 tablet_service.cc:1468] Processing CreateTablet for tablet 2b32e93c114645eeb4e707778c91e12e (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250831 07:58:08.932945 31864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b32e93c114645eeb4e707778c91e12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:09.317940 31946 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:10.293587 31724 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57666 (request call id 56) took 1526 ms. Trace:
I20250831 07:58:10.293931 31724 rpcz_store.cc:276] 0831 07:58:08.768097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.892104 (+124007us) service_pool.cc:225] Handling call
0831 07:58:08.912217 (+ 20113us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.912329 (+ 112us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.287356 (+1375027us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.293204 (+ 5848us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.293540 (+ 336us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.313771 31867 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 42) took 1793 ms. Trace:
I20250831 07:58:10.314152 31867 rpcz_store.cc:276] 0831 07:58:08.520349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.649914 (+129565us) service_pool.cc:225] Handling call
0831 07:58:08.651640 (+ 1726us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.651740 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.305652 (+1653912us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.313217 (+ 7565us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.313702 (+ 485us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.315936 31867 tablet_service.cc:1468] Processing CreateTablet for tablet 6b2e336f43934f118ecacdc1e1d2da60 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250831 07:58:10.317297 31867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b2e336f43934f118ecacdc1e1d2da60. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 07:58:10.318903 31663 debug-util.cc:398] Leaking SignalData structure 0x7b080018e800 after lost signal to thread 30567
W20250831 07:58:10.320122 31663 debug-util.cc:398] Leaking SignalData structure 0x7b080018eee0 after lost signal to thread 31666
I20250831 07:58:10.321823 31950 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:10.322207 31950 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.323771 31950 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:10.332141 31867 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 51) took 1629 ms. Trace:
I20250831 07:58:10.332564 31867 rpcz_store.cc:276] 0831 07:58:08.705465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:10.315034 (+1609569us) service_pool.cc:225] Handling call
0831 07:58:10.316908 (+ 1874us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:10.316992 (+ 84us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.324977 (+ 7985us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.331575 (+ 6598us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.332076 (+ 501us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.334451 31867 tablet_service.cc:1468] Processing CreateTablet for tablet 409f8ff1108b412382695bc6ce64e3cc (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250831 07:58:10.335294 31864 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 50) took 1652 ms. Trace:
I20250831 07:58:10.335657 31867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 409f8ff1108b412382695bc6ce64e3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:10.335676 31864 rpcz_store.cc:276] 0831 07:58:08.682603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.930445 (+247842us) service_pool.cc:225] Handling call
0831 07:58:08.932332 (+ 1887us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.932547 (+ 215us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.328500 (+1395953us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.334793 (+ 6293us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.335238 (+ 445us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.339164 31864 tablet_service.cc:1468] Processing CreateTablet for tablet 26525016efb247c3b1c2bf7fec0e1658 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250831 07:58:10.339217 31871 tablet_service.cc:1468] Processing CreateTablet for tablet 953dd19ad1e4414ab1c956437fd8bbfa (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250831 07:58:10.340512 31864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26525016efb247c3b1c2bf7fec0e1658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:10.341123 31871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 953dd19ad1e4414ab1c956437fd8bbfa. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 07:58:10.347263 31718 consensus_meta.cc:287] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 1.524s user 0.009s sys 0.015s
I20250831 07:58:10.348400 31718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57666 (request call id 46) took 1830 ms. Trace:
I20250831 07:58:10.348773 31718 rpcz_store.cc:276] 0831 07:58:08.517945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.601896 (+ 83951us) service_pool.cc:225] Handling call
0831 07:58:08.684235 (+ 82339us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.684340 (+ 105us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:08.823111 (+138771us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.347870 (+1524759us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.348343 (+ 473us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.352872 31867 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 52) took 1646 ms. Trace:
I20250831 07:58:10.353276 31867 rpcz_store.cc:276] 0831 07:58:08.706943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:10.333516 (+1626573us) service_pool.cc:225] Handling call
0831 07:58:10.335224 (+ 1708us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:10.335309 (+ 85us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.344598 (+ 9289us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.352332 (+ 7734us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.352808 (+ 476us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.355225 31867 tablet_service.cc:1468] Processing CreateTablet for tablet 91390ee2f1e2445ca47e8cb7bd8eeecd (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250831 07:58:10.356503 31629 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59674 (request call id 6) took 1449 ms. Trace:
I20250831 07:58:10.356882 31629 rpcz_store.cc:276] 0831 07:58:08.907404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.918999 (+ 11595us) service_pool.cc:225] Handling call
0831 07:58:10.356398 (+1437399us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20250831 07:58:10.357544 31864 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 53) took 1644 ms. Trace:
W20250831 07:58:10.358194 31680 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 07:58:10.357899 31864 rpcz_store.cc:276] 0831 07:58:08.712720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:10.337899 (+1625179us) service_pool.cc:225] Handling call
0831 07:58:10.340072 (+ 2173us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:10.340164 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.348861 (+ 8697us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.357013 (+ 8152us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.357489 (+ 476us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.358736 31630 rpcz_store.cc:275] Call kudu.master.MasterService.IsCreateTableDone from 127.0.0.1:59680 (request call id 28) took 1051 ms. Trace:
I20250831 07:58:10.359092 31630 rpcz_store.cc:276] 0831 07:58:09.309190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:10.297944 (+988754us) service_pool.cc:225] Handling call
0831 07:58:10.298267 (+ 323us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0831 07:58:10.358404 (+ 60137us) catalog_manager.cc:2266] Verify if the table creation is in progress for test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]
0831 07:58:10.358516 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59919}
I20250831 07:58:10.359941 31864 tablet_service.cc:1468] Processing CreateTablet for tablet 3eb1971f5bb14c27bdbad34649f50ab4 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250831 07:58:10.356768 31867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91390ee2f1e2445ca47e8cb7bd8eeecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:10.361197 31864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eb1971f5bb14c27bdbad34649f50ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:10.361415 31871 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 54) took 1626 ms. Trace:
I20250831 07:58:10.361756 31871 rpcz_store.cc:276] 0831 07:58:08.734629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:10.338281 (+1603652us) service_pool.cc:225] Handling call
0831 07:58:10.340272 (+ 1991us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:10.340360 (+ 88us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.353125 (+ 12765us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.360835 (+ 7710us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.361358 (+ 523us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.363489 31871 tablet_service.cc:1468] Processing CreateTablet for tablet a98f72c022fe465da362553182e4ecca (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250831 07:58:10.364737 31871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98f72c022fe465da362553182e4ecca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:10.368602 31925 tablet_bootstrap.cc:492] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:10.369093 31925 ts_tablet_manager.cc:1397] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 1.570s user 0.008s sys 0.033s
I20250831 07:58:10.371807 31925 raft_consensus.cc:357] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.373975 31925 raft_consensus.cc:383] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.374665 31925 raft_consensus.cc:738] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.375559 31867 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 55) took 1623 ms. Trace:
I20250831 07:58:10.375980 31867 rpcz_store.cc:276] 0831 07:58:08.752287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:10.354258 (+1601971us) service_pool.cc:225] Handling call
0831 07:58:10.356165 (+ 1907us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:10.356283 (+ 118us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.368661 (+ 12378us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.375074 (+ 6413us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.375497 (+ 423us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.375867 31925 consensus_queue.cc:260] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.377768 31867 tablet_service.cc:1468] Processing CreateTablet for tablet fe2bab0189fe475cb540bdf7a041d8b8 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250831 07:58:10.378394 31925 ts_tablet_manager.cc:1428] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.009s user 0.001s sys 0.007s
I20250831 07:58:10.378852 31867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe2bab0189fe475cb540bdf7a041d8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:10.379508 31925 tablet_bootstrap.cc:492] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:10.381384 31864 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 56) took 1601 ms. Trace:
I20250831 07:58:10.381772 31864 rpcz_store.cc:276] 0831 07:58:08.779581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:10.358967 (+1579386us) service_pool.cc:225] Handling call
0831 07:58:10.360757 (+ 1790us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:10.360848 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.368661 (+ 7813us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.380911 (+ 12250us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.381327 (+ 416us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.386495 31925 tablet_bootstrap.cc:654] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.387373 31871 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 57) took 1542 ms. Trace:
I20250831 07:58:10.387753 31871 rpcz_store.cc:276] 0831 07:58:08.844831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:10.362620 (+1517789us) service_pool.cc:225] Handling call
0831 07:58:10.364274 (+ 1654us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:10.364370 (+ 96us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.373511 (+ 9141us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.386885 (+ 13374us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.387316 (+ 431us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.391199 31867 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 58) took 1546 ms. Trace:
I20250831 07:58:10.391909 31925 tablet_bootstrap.cc:492] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:10.391474 31867 rpcz_store.cc:276] 0831 07:58:08.845151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:10.376961 (+1531810us) service_pool.cc:225] Handling call
0831 07:58:10.378444 (+ 1483us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:10.378524 (+ 80us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.385436 (+ 6912us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.390882 (+ 5446us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.391157 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.391683 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:10.392444 31925 ts_tablet_manager.cc:1397] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250831 07:58:10.392534 31808 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:10.394440 31848 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:10.395213 31950 raft_consensus.cc:2802] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:10.395565 31950 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:10.395936 31950 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.403944 31950 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.404498 31870 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 46) took 1882 ms. Trace:
I20250831 07:58:10.404933 31870 rpcz_store.cc:276] 0831 07:58:08.522453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.817469 (+295016us) service_pool.cc:225] Handling call
0831 07:58:08.871904 (+ 54435us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.872006 (+ 102us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.396295 (+1524289us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.402389 (+ 6094us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.404441 (+ 2052us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.406613 31925 raft_consensus.cc:357] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.407359 31925 raft_consensus.cc:383] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.407652 31925 raft_consensus.cc:738] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.408375 31925 consensus_queue.cc:260] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.409590 31790 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39068 (request call id 57) took 1572 ms. Trace:
I20250831 07:58:10.409968 31790 rpcz_store.cc:276] 0831 07:58:08.836941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.837068 (+ 127us) service_pool.cc:225] Handling call
0831 07:58:08.838674 (+ 1606us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.838808 (+ 134us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.403193 (+1564385us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.409245 (+ 6052us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.409536 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.410908 31925 ts_tablet_manager.cc:1428] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20250831 07:58:10.411962 31925 tablet_bootstrap.cc:492] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:10.413905 31950 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:10.404320 31868 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 48) took 1816 ms. Trace:
I20250831 07:58:10.414136 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:10.414767 31808 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.414697 31868 rpcz_store.cc:276] 0831 07:58:08.588088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.860562 (+272474us) service_pool.cc:225] Handling call
0831 07:58:08.862583 (+ 2021us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.862682 (+ 99us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.396589 (+1533907us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.403758 (+ 7169us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.404263 (+ 505us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.414799 31865 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 41) took 2060 ms. Trace:
I20250831 07:58:10.417263 31865 rpcz_store.cc:276] 0831 07:58:08.353953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.539574 (+185621us) service_pool.cc:225] Handling call
0831 07:58:08.611404 (+ 71830us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.611491 (+ 87us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:08.850551 (+239060us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:08.924077 (+ 73526us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.414463 (+1490386us) spinlock_profiling.cc:243] Waited 4.05 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e882 00007fdcedd81b29 00007fdcecee4ea8 00007fdcecfa9773 00007fdcecfa553f 00007fdcecfa5cfd 00007fdcecf56209 00007fdce6505bf4 00007fdce6505b82
0831 07:58:10.414743 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":344,"spinlock_wait_cycles":9314688}
I20250831 07:58:10.418766 31925 tablet_bootstrap.cc:654] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
W20250831 07:58:10.421725 31721 consensus_meta.cc:287] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 1.627s user 0.014s sys 0.013s
I20250831 07:58:10.422484 31721 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57666 (request call id 49) took 1827 ms. Trace:
I20250831 07:58:10.422755 31721 rpcz_store.cc:276] 0831 07:58:08.594914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.703335 (+108421us) service_pool.cc:225] Handling call
0831 07:58:08.735438 (+ 32103us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.735670 (+ 232us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:08.794084 (+ 58414us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.422126 (+1628042us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.422443 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.425381 31808 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
W20250831 07:58:10.428119 31795 consensus_meta.cc:287] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 1.625s user 0.024s sys 0.025s
I20250831 07:58:10.429476 31795 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39068 (request call id 53) took 1718 ms. Trace:
I20250831 07:58:10.430034 31795 rpcz_store.cc:276] 0831 07:58:08.711092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.714916 (+ 3824us) service_pool.cc:225] Handling call
0831 07:58:08.739931 (+ 25015us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.740042 (+ 111us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:08.802933 (+ 62891us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.428800 (+1625867us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.429404 (+ 604us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.432406 31866 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 47) took 1896 ms. Trace:
I20250831 07:58:10.432765 31866 rpcz_store.cc:276] 0831 07:58:08.536253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.824206 (+287953us) service_pool.cc:225] Handling call
0831 07:58:08.890239 (+ 66033us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.890356 (+ 117us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.424612 (+1534256us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.431912 (+ 7300us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.432352 (+ 440us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.437252 31848 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:10.438127 31950 raft_consensus.cc:2802] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:10.439908 31925 tablet_bootstrap.cc:492] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:10.440393 31925 ts_tablet_manager.cc:1397] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.009s
I20250831 07:58:10.443588 31950 raft_consensus.cc:695] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:10.443377 31925 raft_consensus.cc:357] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.444931 31925 raft_consensus.cc:383] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.444600 31950 consensus_queue.cc:237] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:10.445050 31715 consensus_meta.cc:287] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 1.522s user 0.012s sys 0.000s
I20250831 07:58:10.445254 31925 raft_consensus.cc:738] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.446203 31715 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57666 (request call id 48) took 1905 ms. Trace:
I20250831 07:58:10.446090 31925 consensus_queue.cc:260] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.446578 31715 rpcz_store.cc:276] 0831 07:58:08.541012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.668234 (+127222us) service_pool.cc:225] Handling call
0831 07:58:08.669908 (+ 1674us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.669999 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:08.922939 (+252940us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.445659 (+1522720us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.446141 (+ 482us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.448493 31925 ts_tablet_manager.cc:1428] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250831 07:58:10.450798 31925 tablet_bootstrap.cc:492] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:08.923408 31719 tablet_service.cc:1468] Processing CreateTablet for tablet fe2bab0189fe475cb540bdf7a041d8b8 (DEFAULT_TABLE table=test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250831 07:58:10.456117 31719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe2bab0189fe475cb540bdf7a041d8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:10.456029 31869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 43) took 1935 ms. Trace:
W20250831 07:58:10.456267 31720 consensus_meta.cc:287] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 1.629s user 0.030s sys 0.007s
I20250831 07:58:10.456643 31869 rpcz_store.cc:276] 0831 07:58:08.520718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.557854 (+ 37136us) service_pool.cc:225] Handling call
0831 07:58:08.567920 (+ 10066us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.568252 (+ 332us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.446760 (+1878508us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.455492 (+ 8732us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.455969 (+ 477us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.460760 31925 tablet_bootstrap.cc:654] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.466797 31924 raft_consensus.cc:357] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.467538 31924 raft_consensus.cc:383] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.467862 31924 raft_consensus.cc:738] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.467967 31720 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57666 (request call id 45) took 1988 ms. Trace:
I20250831 07:58:10.469110 31924 consensus_queue.cc:260] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.468331 31720 rpcz_store.cc:276] 0831 07:58:08.479626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.586522 (+106896us) service_pool.cc:225] Handling call
0831 07:58:08.661167 (+ 74645us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.661258 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:08.826885 (+165627us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.456858 (+1629973us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.467895 (+ 11037us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.472915 31952 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:10.473407 31952 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.475701 31923 tablet_bootstrap.cc:492] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:10.476006 31952 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.476222 31923 ts_tablet_manager.cc:1397] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 1.656s user 0.012s sys 0.027s
I20250831 07:58:10.477743 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:10.479112 31923 raft_consensus.cc:357] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:10.479545 31773 leader_election.cc:343] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.480417 31719 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57666 (request call id 58) took 1635 ms. Trace:
I20250831 07:58:10.480844 31719 rpcz_store.cc:276] 0831 07:58:08.845261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.922098 (+ 76837us) service_pool.cc:225] Handling call
0831 07:58:10.455641 (+1533543us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:10.455736 (+ 95us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.466542 (+ 10806us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.476381 (+ 9839us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.480363 (+ 3982us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.481858 31924 ts_tablet_manager.cc:1428] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 1.555s user 0.017s sys 0.010s
W20250831 07:58:10.482167 31924 ts_tablet_manager.cc:1460] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c: Tablet startup took 1555ms
I20250831 07:58:10.482508 31717 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57666 (request call id 57) took 1701 ms. Trace:
I20250831 07:58:10.482930 31717 rpcz_store.cc:276] 0831 07:58:08.781492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.902925 (+121433us) service_pool.cc:225] Handling call
0831 07:58:08.907459 (+ 4534us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.907548 (+ 89us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.463246 (+1555698us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.478347 (+ 15101us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.482452 (+ 4105us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:10.482470 31924 ts_tablet_manager.cc:1462] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c: Trace:
0831 07:58:07.634933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:07.647638 (+ 12705us) service_pool.cc:225] Handling call
0831 07:58:07.649862 (+ 2224us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:07.649992 (+ 130us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:07.703267 (+ 53275us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:07.721309 (+ 18042us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:07.721704 (+ 395us) inbound_call.cc:173] Queueing success response
0831 07:58:08.654608 (+932904us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0831 07:58:08.926756 (+272148us) ts_tablet_manager.cc:1429] Starting tablet replica
0831 07:58:08.927298 (+ 542us) tablet_replica.cc:260] Starting instrumentation
0831 07:58:09.209325 (+282027us) tablet_replica.cc:279] Starting consensus
Metrics: {"raft.queue_time_us":9177,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":430,"tablet-open.queue_time_us":932966,"thread_start_us":10238,"threads_started":1}
I20250831 07:58:10.485042 31925 tablet_bootstrap.cc:492] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:10.485497 31925 ts_tablet_manager.cc:1397] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.011s
I20250831 07:58:10.487190 31924 tablet_bootstrap.cc:492] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:10.493358 31925 raft_consensus.cc:357] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.494043 31925 raft_consensus.cc:383] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.494313 31925 raft_consensus.cc:738] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.495059 31952 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:10.494927 31925 consensus_queue.cc:260] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.495540 31952 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.497009 31925 ts_tablet_manager.cc:1428] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20250831 07:58:10.497815 31925 tablet_bootstrap.cc:492] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:10.499215 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:10.495442 31923 raft_consensus.cc:383] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.500032 31923 raft_consensus.cc:738] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.501192 31716 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57666 (request call id 55) took 1757 ms. Trace:
I20250831 07:58:10.501464 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:58:10.502102 31723 consensus_meta.cc:287] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 1.591s user 0.010s sys 0.012s
I20250831 07:58:10.502130 31809 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:10.502612 31773 leader_election.cc:343] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.500716 31952 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.504715 31952 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:10.505348 31952 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.505770 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:10.508284 31775 leader_election.cc:343] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.508729 31775 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:10.508919 31923 consensus_queue.cc:260] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.513881 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:10.514456 31883 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:10.503268 31723 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57666 (request call id 47) took 1982 ms. Trace:
I20250831 07:58:10.514998 31723 rpcz_store.cc:276] 0831 07:58:08.521161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.643381 (+122220us) service_pool.cc:225] Handling call
0831 07:58:08.788975 (+145594us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.789075 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:08.910538 (+121463us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.502774 (+1592236us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.503215 (+ 441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.509966 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:10.516646 31734 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:10.510084 31952 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.501598 31716 rpcz_store.cc:276] 0831 07:58:08.744001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.881026 (+137025us) service_pool.cc:225] Handling call
0831 07:58:08.888262 (+ 7236us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.888362 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.482806 (+1594444us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.499376 (+ 16570us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.501138 (+ 1762us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.513216 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:10.517421 31952 raft_consensus.cc:2747] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:10.517925 31923 ts_tablet_manager.cc:1428] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.041s user 0.008s sys 0.000s
I20250831 07:58:10.534211 31923 tablet_bootstrap.cc:492] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:10.518649 31773 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:10.536342 31952 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:10.536720 31952 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:10.537074 31952 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.537935 31872 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 44) took 2016 ms. Trace:
I20250831 07:58:10.538363 31872 rpcz_store.cc:276] 0831 07:58:08.521048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.746034 (+224986us) service_pool.cc:225] Handling call
0831 07:58:08.768772 (+ 22738us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.776670 (+ 7898us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.528965 (+1752295us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.537425 (+ 8460us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.537881 (+ 456us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.517724 31775 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:10.540522 31929 raft_consensus.cc:2802] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:10.522080 31924 tablet_bootstrap.cc:654] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.522081 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:10.540896 31929 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:10.541242 31929 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.521575 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:10.532083 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:58:10.539872 31663 debug-util.cc:398] Leaking SignalData structure 0x7b08001104a0 after lost signal to thread 31758
W20250831 07:58:10.543159 31849 leader_election.cc:343] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:10.543273 31663 debug-util.cc:398] Leaking SignalData structure 0x7b08000984c0 after lost signal to thread 31833
I20250831 07:58:10.520375 31809 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
W20250831 07:58:10.544361 31775 leader_election.cc:343] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.545944 31848 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:10.546844 31950 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:10.547360 31950 raft_consensus.cc:695] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:10.548414 31950 consensus_queue.cc:237] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:10.549940 31952 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:10.544361 31663 debug-util.cc:398] Leaking SignalData structure 0x7b0800204ea0 after lost signal to thread 31907
W20250831 07:58:10.551132 31663 debug-util.cc:398] Leaking SignalData structure 0x7b0800204da0 after lost signal to thread 31922
W20250831 07:58:10.551498 31663 debug-util.cc:398] Leaking SignalData structure 0x7b0800204d40 after lost signal to thread 31928
W20250831 07:58:10.551904 31663 debug-util.cc:398] Leaking SignalData structure 0x7b0800204ca0 after lost signal to thread 31943
I20250831 07:58:10.553550 31929 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.557268 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:10.558413 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:10.559474 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:58:10.559682 31773 leader_election.cc:343] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.560250 31734 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.562651 31791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39068 (request call id 56) took 1782 ms. Trace:
I20250831 07:58:10.563092 31958 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:10.563915 31929 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:10.544111 31849 leader_election.cc:343] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.563849 31958 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.569176 31958 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.570071 31925 tablet_bootstrap.cc:654] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.570289 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:10.571184 31923 tablet_bootstrap.cc:654] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
W20250831 07:58:10.571676 31773 leader_election.cc:343] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.563066 31791 rpcz_store.cc:276] 0831 07:58:08.779870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.782045 (+ 2175us) service_pool.cc:225] Handling call
0831 07:58:08.837766 (+ 55721us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.837850 (+ 84us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.527068 (+1689218us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.562148 (+ 35080us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.562592 (+ 444us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:10.576046 31775 leader_election.cc:343] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.576403 31628 catalog_manager.cc:5582] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:10.577689 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:10.579284 31775 leader_election.cc:343] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.581452 31873 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57332 (request call id 49) took 1974 ms. Trace:
I20250831 07:58:10.581557 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:10.581998 31952 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.582243 31883 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:10.574141 31849 leader_election.cc:343] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.589387 31734 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:10.584041 31873 rpcz_store.cc:276] 0831 07:58:08.606845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:08.868485 (+261640us) service_pool.cc:225] Handling call
0831 07:58:08.870306 (+ 1821us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0831 07:58:08.870421 (+ 115us) ts_tablet_manager.cc:676] Creating new metadata...
0831 07:58:10.511602 (+1641181us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:10.580958 (+ 69356us) tablet_replica.cc:206] Creating consensus instance
0831 07:58:10.581400 (+ 442us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:10.593566 31883 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:10.579887 31775 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:10.597769 31773 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:10.598321 31924 tablet_bootstrap.cc:492] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:10.598281 31775 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:10.599050 31924 ts_tablet_manager.cc:1397] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.113s user 0.023s sys 0.040s
I20250831 07:58:10.598821 31958 raft_consensus.cc:2747] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:10.599399 31929 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:10.599850 31929 raft_consensus.cc:695] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:10.599862 31958 raft_consensus.cc:2802] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:10.600489 31958 raft_consensus.cc:695] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:10.600737 31929 consensus_queue.cc:237] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.601253 31958 consensus_queue.cc:237] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.602068 31924 raft_consensus.cc:357] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:10.603484 31957 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:10.602811 31924 raft_consensus.cc:383] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.604008 31957 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.604434 31924 raft_consensus.cc:738] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.606036 31924 consensus_queue.cc:260] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:10.607051 31957 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.611224 31923 tablet_bootstrap.cc:492] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:10.611892 31923 ts_tablet_manager.cc:1397] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.078s user 0.007s sys 0.007s
I20250831 07:58:10.613435 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:10.616328 31925 tablet_bootstrap.cc:492] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:10.616505 31883 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:10.616791 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:10.617115 31925 ts_tablet_manager.cc:1397] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.119s user 0.012s sys 0.001s
W20250831 07:58:10.617940 31699 leader_election.cc:343] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.620586 31698 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:10.621908 31957 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:10.622280 31957 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:10.623344 31957 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.624359 31925 raft_consensus.cc:357] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.625288 31925 raft_consensus.cc:383] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.625635 31925 raft_consensus.cc:738] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.626441 31925 consensus_queue.cc:260] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.627274 31952 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:10.627746 31952 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.630059 31952 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.631930 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:10.633114 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:10.633227 31773 leader_election.cc:343] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:10.634814 31775 leader_election.cc:343] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.635329 31775 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:10.636593 31929 raft_consensus.cc:2747] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:10.637324 31924 ts_tablet_manager.cc:1428] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.038s user 0.014s sys 0.018s
I20250831 07:58:10.638986 31924 tablet_bootstrap.cc:492] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:10.622110 31923 raft_consensus.cc:357] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.640066 31923 raft_consensus.cc:383] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.640448 31923 raft_consensus.cc:738] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.641207 31957 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.641300 31923 consensus_queue.cc:260] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.644726 31923 ts_tablet_manager.cc:1428] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.032s user 0.004s sys 0.004s
I20250831 07:58:10.646270 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:10.647794 31924 tablet_bootstrap.cc:654] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.652467 31923 tablet_bootstrap.cc:492] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
W20250831 07:58:10.652688 31699 leader_election.cc:343] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.648451 31925 ts_tablet_manager.cc:1428] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.031s user 0.015s sys 0.012s
I20250831 07:58:10.656348 31630 catalog_manager.cc:5582] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:10.661010 31923 tablet_bootstrap.cc:654] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.656394 31925 tablet_bootstrap.cc:492] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:10.668174 31924 tablet_bootstrap.cc:492] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:10.648838 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:10.672530 31883 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.672652 31924 ts_tablet_manager.cc:1397] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.005s
I20250831 07:58:10.673209 31967 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:10.648538 31957 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.673681 31967 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.675844 31924 raft_consensus.cc:357] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.676709 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:10.680178 31883 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:10.680426 31924 raft_consensus.cc:383] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.680615 31734 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:10.680801 31924 raft_consensus.cc:738] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.678565 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:10.681588 31924 consensus_queue.cc:260] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.682848 31698 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:10.684026 31957 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:10.684509 31957 raft_consensus.cc:695] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:10.684628 31924 ts_tablet_manager.cc:1428] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250831 07:58:10.685895 31924 tablet_bootstrap.cc:492] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:10.685412 31957 consensus_queue.cc:237] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:10.690250 31909 tablet.cc:2378] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250831 07:58:10.691658 31773 leader_election.cc:343] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.675767 31925 tablet_bootstrap.cc:654] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.679044 31967 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.693928 31966 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:10.694242 31775 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:10.694658 31966 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.695355 31967 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:10.695734 31924 tablet_bootstrap.cc:654] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.695756 31967 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:10.696620 31967 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.704275 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:10.705281 31966 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.705756 31809 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:10.708415 31699 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:10.709306 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:10.714618 31882 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:10.714941 31925 tablet_bootstrap.cc:492] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:10.715368 31925 ts_tablet_manager.cc:1397] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.059s user 0.012s sys 0.003s
I20250831 07:58:10.716470 31957 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:10.716979 31957 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:10.717410 31957 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.717804 31925 raft_consensus.cc:357] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.718456 31925 raft_consensus.cc:383] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.718719 31925 raft_consensus.cc:738] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.719446 31925 consensus_queue.cc:260] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.719602 31967 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.722325 31967 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.707909 31923 tablet_bootstrap.cc:492] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:10.723397 31923 ts_tablet_manager.cc:1397] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.071s user 0.013s sys 0.009s
I20250831 07:58:10.723989 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:10.722676 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:10.710259 31924 tablet_bootstrap.cc:492] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:10.724804 31734 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.727790 31924 ts_tablet_manager.cc:1397] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.001s
I20250831 07:58:10.730676 31924 raft_consensus.cc:357] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.732479 31924 raft_consensus.cc:383] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.732844 31924 raft_consensus.cc:738] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.734134 31924 consensus_queue.cc:260] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.727440 31923 raft_consensus.cc:357] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.736750 31923 raft_consensus.cc:383] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.737773 31923 raft_consensus.cc:738] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.737411 31957 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.738648 31923 consensus_queue.cc:260] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.740399 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:10.740993 31809 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.742180 31734 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:10.743332 31775 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:10.744100 31923 ts_tablet_manager.cc:1428] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.020s user 0.007s sys 0.001s
I20250831 07:58:10.745750 31923 tablet_bootstrap.cc:492] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:10.746156 31967 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:10.748080 31967 raft_consensus.cc:695] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:10.748910 31967 consensus_queue.cc:237] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.749361 31809 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:10.750308 31699 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:10.751439 31972 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:10.740556 31882 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.744647 31924 ts_tablet_manager.cc:1428] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.016s user 0.004s sys 0.003s
I20250831 07:58:10.749362 31631 catalog_manager.cc:5582] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:10.752875 31972 raft_consensus.cc:695] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:10.755601 31972 consensus_queue.cc:237] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.753677 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:10.753994 31957 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.758854 31883 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.759955 31925 ts_tablet_manager.cc:1428] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.044s user 0.008s sys 0.000s
I20250831 07:58:10.760659 31925 tablet_bootstrap.cc:492] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:10.768132 31924 tablet_bootstrap.cc:492] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:10.768429 31882 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:10.773322 31883 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:10.775750 31925 tablet_bootstrap.cc:654] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.783032 31631 catalog_manager.cc:5582] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:10.786128 31923 tablet_bootstrap.cc:654] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.790232 31924 tablet_bootstrap.cc:654] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.797350 31628 catalog_manager.cc:5582] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:10.799422 31630 catalog_manager.cc:5582] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:10.803380 31925 tablet_bootstrap.cc:492] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:10.805141 31925 ts_tablet_manager.cc:1397] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.003s
I20250831 07:58:10.805482 31630 catalog_manager.cc:5582] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:10.812121 31925 raft_consensus.cc:357] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.818552 31925 raft_consensus.cc:383] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.818775 31925 raft_consensus.cc:738] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.819365 31925 consensus_queue.cc:260] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.823084 31925 ts_tablet_manager.cc:1428] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250831 07:58:10.824118 31925 tablet_bootstrap.cc:492] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:10.831151 31925 tablet_bootstrap.cc:654] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.834172 31924 tablet_bootstrap.cc:492] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:10.834765 31924 ts_tablet_manager.cc:1397] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.067s user 0.026s sys 0.013s
I20250831 07:58:10.837250 31924 raft_consensus.cc:357] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.837796 31924 raft_consensus.cc:383] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.838050 31924 raft_consensus.cc:738] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.838653 31924 consensus_queue.cc:260] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.841379 31924 ts_tablet_manager.cc:1428] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250831 07:58:10.841817 31923 tablet_bootstrap.cc:492] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:10.842288 31923 ts_tablet_manager.cc:1397] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.097s user 0.017s sys 0.015s
I20250831 07:58:10.845079 31924 tablet_bootstrap.cc:492] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:10.846256 31923 raft_consensus.cc:357] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.847047 31923 raft_consensus.cc:383] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.847368 31923 raft_consensus.cc:738] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.848165 31923 consensus_queue.cc:260] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.852445 31924 tablet_bootstrap.cc:654] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.856104 31925 tablet_bootstrap.cc:492] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:10.856582 31925 ts_tablet_manager.cc:1397] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.007s
I20250831 07:58:10.859189 31925 raft_consensus.cc:357] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.859972 31925 raft_consensus.cc:383] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.860261 31925 raft_consensus.cc:738] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.861299 31925 consensus_queue.cc:260] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.864781 31923 ts_tablet_manager.cc:1428] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
I20250831 07:58:10.873164 31925 ts_tablet_manager.cc:1428] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250831 07:58:10.876211 31923 tablet_bootstrap.cc:492] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:10.878103 31925 tablet_bootstrap.cc:492] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:10.880538 31967 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:10.881048 31967 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.883105 31967 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.884793 31925 tablet_bootstrap.cc:654] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.885404 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:10.886693 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:10.888149 31775 leader_election.cc:343] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:10.889518 31773 leader_election.cc:343] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.890089 31773 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:10.891614 31967 raft_consensus.cc:2747] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:10.892508 31923 tablet_bootstrap.cc:654] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.893337 31924 tablet_bootstrap.cc:492] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:10.893903 31924 ts_tablet_manager.cc:1397] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.049s user 0.017s sys 0.028s
I20250831 07:58:10.896867 31924 raft_consensus.cc:357] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.897728 31924 raft_consensus.cc:383] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.898054 31924 raft_consensus.cc:738] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.898957 31924 consensus_queue.cc:260] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.901526 31924 ts_tablet_manager.cc:1428] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250831 07:58:10.902432 31924 tablet_bootstrap.cc:492] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:10.906461 31925 tablet_bootstrap.cc:492] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:10.906952 31925 ts_tablet_manager.cc:1397] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.019s
I20250831 07:58:10.907171 31972 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:10.907635 31972 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.908236 31924 tablet_bootstrap.cc:654] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.909591 31925 raft_consensus.cc:357] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.910339 31925 raft_consensus.cc:383] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.910600 31925 raft_consensus.cc:738] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.911232 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:10.911362 31925 consensus_queue.cc:260] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.914064 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:10.914862 31809 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
W20250831 07:58:10.917210 31698 leader_election.cc:343] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:10.918215 31925 ts_tablet_manager.cc:1428] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250831 07:58:10.918355 31699 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:10.919260 31925 tablet_bootstrap.cc:492] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:10.919903 31957 raft_consensus.cc:2802] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:10.920414 31957 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:10.920864 31957 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.909662 31972 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.926260 31925 tablet_bootstrap.cc:654] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.926405 31924 tablet_bootstrap.cc:492] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:10.926815 31924 ts_tablet_manager.cc:1397] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.007s
I20250831 07:58:10.929325 31924 raft_consensus.cc:357] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.930186 31924 raft_consensus.cc:383] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.930521 31924 raft_consensus.cc:738] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.935776 31924 consensus_queue.cc:260] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.938951 31924 ts_tablet_manager.cc:1428] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20250831 07:58:10.936034 31923 tablet_bootstrap.cc:492] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:10.941174 31923 ts_tablet_manager.cc:1397] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.065s user 0.019s sys 0.000s
I20250831 07:58:10.943902 31923 raft_consensus.cc:357] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.944543 31923 raft_consensus.cc:383] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.944813 31923 raft_consensus.cc:738] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.945979 31924 tablet_bootstrap.cc:492] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:10.952065 31923 consensus_queue.cc:260] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.954232 31923 ts_tablet_manager.cc:1428] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20250831 07:58:10.956198 31924 tablet_bootstrap.cc:654] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:10.957161 31972 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:10.957644 31972 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.960800 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:10.958182 31923 tablet_bootstrap.cc:492] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:10.963331 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:10.964102 31809 raft_consensus.cc:2391] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:10.964982 31957 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.965512 31699 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:10.966002 31972 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:10.968015 31925 tablet_bootstrap.cc:492] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:10.968466 31925 ts_tablet_manager.cc:1397] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.049s user 0.024s sys 0.013s
I20250831 07:58:10.971630 31925 raft_consensus.cc:357] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.972360 31925 raft_consensus.cc:383] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:10.972736 31925 raft_consensus.cc:738] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:10.973368 31925 consensus_queue.cc:260] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:10.974849 31950 consensus_queue.cc:1035] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:10.980180 31966 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:10.980849 31966 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:10.985679 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:10.987423 31966 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:10.986058 31925 ts_tablet_manager.cc:1428] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20250831 07:58:10.967121 31972 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:10.988783 31925 tablet_bootstrap.cc:492] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:10.991168 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:10.992863 31699 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:10.993235 31924 tablet_bootstrap.cc:492] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:10.993752 31924 ts_tablet_manager.cc:1397] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.048s user 0.018s sys 0.019s
I20250831 07:58:10.993958 31966 raft_consensus.cc:2747] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:11.001230 31925 tablet_bootstrap.cc:654] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.003878 31923 tablet_bootstrap.cc:654] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.004379 31924 raft_consensus.cc:357] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.005059 31924 raft_consensus.cc:383] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.005302 31924 raft_consensus.cc:738] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.006031 31924 consensus_queue.cc:260] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.008292 31924 ts_tablet_manager.cc:1428] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.014s user 0.006s sys 0.002s
I20250831 07:58:11.009364 31924 tablet_bootstrap.cc:492] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
W20250831 07:58:11.016244 31773 consensus_peers.cc:489] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 -> Peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Couldn't send request to peer 8247a38b550649aa9cd6e7668dcaec10. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:11.017079 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:11.017297 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:11.017773 31808 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.018085 31882 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.025358 31957 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.029560 31808 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:11.032191 31734 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.032575 31957 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.038512 31957 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.041236 31924 tablet_bootstrap.cc:654] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.043505 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:11.027257 31972 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:11.031383 31699 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:58:11.047673 31698 leader_election.cc:343] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.045611 31957 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.048718 31972 raft_consensus.cc:2802] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:11.051118 31882 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:11.053562 31923 tablet_bootstrap.cc:492] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:11.055927 31925 tablet_bootstrap.cc:492] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:11.059283 31923 ts_tablet_manager.cc:1397] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.101s user 0.011s sys 0.018s
I20250831 07:58:11.060124 31925 ts_tablet_manager.cc:1397] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.072s user 0.044s sys 0.006s
I20250831 07:58:11.063535 31925 raft_consensus.cc:357] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.064145 31925 raft_consensus.cc:383] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.064320 31925 raft_consensus.cc:738] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.058320 31972 raft_consensus.cc:695] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:11.068099 31925 consensus_queue.cc:260] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.068437 31972 consensus_queue.cc:237] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.070201 31925 ts_tablet_manager.cc:1428] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250831 07:58:11.071581 31923 raft_consensus.cc:357] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.072585 31923 raft_consensus.cc:383] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.072865 31923 raft_consensus.cc:738] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.073776 31923 consensus_queue.cc:260] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.076853 31950 consensus_queue.cc:1035] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:11.079496 31925 tablet_bootstrap.cc:492] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:11.085340 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:11.086073 31809 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:11.089931 31954 consensus_queue.cc:1035] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250831 07:58:11.090606 31699 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:11.089485 31967 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.091670 31967 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.092104 31957 raft_consensus.cc:2802] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:11.093838 31808 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.094449 31967 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.092595 31957 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:11.095744 31957 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.096697 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:11.097327 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:11.098114 31775 leader_election.cc:343] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.098487 31966 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 07:58:11.098841 31773 leader_election.cc:343] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.099368 31773 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:11.100598 31954 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.104612 31967 raft_consensus.cc:2747] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:11.107441 31954 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.118993 31923 ts_tablet_manager.cc:1428] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.058s user 0.006s sys 0.001s
I20250831 07:58:11.120134 31923 tablet_bootstrap.cc:492] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:11.126202 31734 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.127027 31924 tablet_bootstrap.cc:492] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:11.127462 31924 ts_tablet_manager.cc:1397] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.118s user 0.008s sys 0.020s
I20250831 07:58:11.127633 31954 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:11.138964 31924 raft_consensus.cc:357] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.139528 31924 raft_consensus.cc:383] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.139763 31924 raft_consensus.cc:738] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.140364 31924 consensus_queue.cc:260] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.141091 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:11.141700 31807 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:11.142521 31924 ts_tablet_manager.cc:1428] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250831 07:58:11.143028 31848 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:11.144945 31924 tablet_bootstrap.cc:492] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:11.148582 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:11.149314 31954 raft_consensus.cc:2802] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:11.150087 31925 tablet_bootstrap.cc:654] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.150317 31954 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:11.150662 31954 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.152948 31950 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.153393 31950 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.155004 31972 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.155416 31972 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.159055 31972 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.157330 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:11.163002 31807 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:11.155799 31950 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:11.164546 31848 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:11.159736 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:11.168481 31924 tablet_bootstrap.cc:654] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.168236 31997 raft_consensus.cc:2802] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:11.159348 31929 consensus_queue.cc:1035] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250831 07:58:11.168831 31997 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:11.168845 31882 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:11.173370 31698 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:11.160434 31999 consensus_queue.cc:1035] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
I20250831 07:58:11.174211 31972 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:11.175385 31972 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:11.175719 31972 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:11.176294 31923 tablet_bootstrap.cc:654] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.169927 31997 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.181174 31954 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.158598 31957 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.171705 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:11.172024 31849 leader_election.cc:343] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.199787 31929 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.200320 31929 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.159437 31628 catalog_manager.cc:5582] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:11.196070 31967 consensus_queue.cc:1035] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.042s
W20250831 07:58:11.262207 31955 log.cc:927] Time spent T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.056s user 0.010s sys 0.000s
I20250831 07:58:11.274875 31925 tablet_bootstrap.cc:492] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:11.275525 31925 ts_tablet_manager.cc:1397] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.196s user 0.028s sys 0.018s
I20250831 07:58:11.289157 32005 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250831 07:58:11.294092 31954 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:11.299492 31966 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.300272 31966 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.302899 31966 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.310830 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:11.314184 31698 leader_election.cc:343] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.320647 31957 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.321336 31957 consensus_queue.cc:1035] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:11.321774 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:11.324059 31972 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.325301 31734 raft_consensus.cc:1238] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250831 07:58:11.327487 31972 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.328505 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:11.329074 31882 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:11.330165 31925 raft_consensus.cc:357] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.330804 31925 raft_consensus.cc:383] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.331082 31925 raft_consensus.cc:738] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.331686 31925 consensus_queue.cc:260] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.279654 31967 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.043s
I20250831 07:58:11.345890 31925 ts_tablet_manager.cc:1428] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.070s user 0.005s sys 0.000s
I20250831 07:58:11.347417 31925 tablet_bootstrap.cc:492] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:11.333536 31929 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.338142 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:58:11.267262 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 97098.6us
W20250831 07:58:11.335630 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 133197us
W20250831 07:58:11.343791 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 154162us
I20250831 07:58:11.359158 32009 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.353478 31809 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.359663 32009 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.347249 31997 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.352638 31929 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.373042 31924 tablet_bootstrap.cc:492] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:11.373706 31924 ts_tablet_manager.cc:1397] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.230s user 0.019s sys 0.025s
I20250831 07:58:11.374186 31883 raft_consensus.cc:2391] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:11.375706 31997 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:11.379707 31924 raft_consensus.cc:357] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.381165 31924 raft_consensus.cc:383] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.381384 31924 raft_consensus.cc:738] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.381860 31924 consensus_queue.cc:260] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.383677 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:11.384646 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:11.385097 32008 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.385200 31807 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:11.385354 31680 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 07:58:11.385674 32008 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.399660 32009 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.401963 31882 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:11.366966 31929 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.392120 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:11.405277 31881 raft_consensus.cc:1238] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250831 07:58:11.423283 31809 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:11.424136 32008 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.446081 31967 consensus_queue.cc:1046] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:11.447467 31929 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.400832 31807 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:11.452661 31924 ts_tablet_manager.cc:1428] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.079s user 0.009s sys 0.000s
I20250831 07:58:11.453264 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:11.454198 31698 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:11.459009 31923 tablet_bootstrap.cc:492] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:11.460289 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:11.460669 31924 tablet_bootstrap.cc:492] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:11.454319 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:11.462390 31923 ts_tablet_manager.cc:1397] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.343s user 0.017s sys 0.011s
I20250831 07:58:11.456689 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:11.465557 31848 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:11.466423 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:11.469393 31731 raft_consensus.cc:2391] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 1: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:11.469213 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:11.467886 31808 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:11.469794 31729 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.472862 31954 raft_consensus.cc:2802] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:11.473191 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:11.473361 31954 raft_consensus.cc:695] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:11.474249 31954 consensus_queue.cc:237] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.479761 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:11.472093 31957 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:11.481529 31957 raft_consensus.cc:695] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:58:11.463229 31775 leader_election.cc:343] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.483274 31957 consensus_queue.cc:237] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.483893 31806 raft_consensus.cc:2391] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:11.494529 32008 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 07:58:11.490736 31773 leader_election.cc:343] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:11.500180 31775 leader_election.cc:343] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.507263 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:11.483948 32004 consensus_queue.cc:1035] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.047s
I20250831 07:58:11.505649 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:11.516710 31848 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:11.525049 31999 raft_consensus.cc:2802] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:11.525732 31999 raft_consensus.cc:695] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:11.526651 31999 consensus_queue.cc:237] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.528440 32008 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.528944 32008 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.513069 31699 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:11.533279 31727 raft_consensus.cc:2391] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 1: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:11.534837 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:11.535586 31923 raft_consensus.cc:357] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.533495 32010 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:11.536862 31923 raft_consensus.cc:383] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.536564 32010 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:11.537165 31923 raft_consensus.cc:738] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.537576 32010 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:11.537596 31775 leader_election.cc:343] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.538194 31923 consensus_queue.cc:260] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.538626 32008 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.541049 31923 ts_tablet_manager.cc:1428] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.078s user 0.005s sys 0.001s
I20250831 07:58:11.535219 31729 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:11.544602 31925 tablet_bootstrap.cc:654] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.524250 32014 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.547848 32009 raft_consensus.cc:3053] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:11.548249 32009 raft_consensus.cc:738] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:11.536501 31699 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:11.549075 32009 consensus_queue.cc:260] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.551390 32009 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:11.545909 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:11.557868 31966 raft_consensus.cc:2747] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250831 07:58:11.574903 31923 tablet_bootstrap.cc:492] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:11.580185 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:11.585235 31773 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:11.560979 32010 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.546948 32014 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.586833 31967 raft_consensus.cc:2747] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 07:58:11.590058 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 114294us
I20250831 07:58:11.601672 31923 tablet_bootstrap.cc:654] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.605106 31925 tablet_bootstrap.cc:492] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:11.605578 31925 ts_tablet_manager.cc:1397] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.259s user 0.008s sys 0.021s
I20250831 07:58:11.607753 31730 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
W20250831 07:58:11.611147 31773 leader_election.cc:343] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.612149 31924 tablet_bootstrap.cc:654] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.612594 31773 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:11.613404 32008 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 07:58:11.614045 31773 leader_election.cc:343] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.614449 31773 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:11.614668 32008 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:11.614960 31775 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:11.615190 31967 raft_consensus.cc:2747] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:11.615845 31967 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:11.616156 31967 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:11.616420 31967 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.616791 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:11.617530 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:11.618906 31698 leader_election.cc:343] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.616531 32014 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.623839 31967 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.624789 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:11.627530 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:11.628136 31730 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.628611 31925 raft_consensus.cc:357] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.629154 31925 raft_consensus.cc:383] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.629348 31925 raft_consensus.cc:738] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.630700 31967 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.632102 31804 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:11.633097 31925 consensus_queue.cc:260] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:11.643954 31773 leader_election.cc:343] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.649724 31925 ts_tablet_manager.cc:1428] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.044s user 0.004s sys 0.004s
I20250831 07:58:11.655086 31925 tablet_bootstrap.cc:492] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:11.654881 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:58:11.659451 31698 leader_election.cc:343] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.661856 31630 catalog_manager.cc:5582] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 1 to 2, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 2 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:11.664711 32014 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:11.650571 31773 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:11.666356 31967 raft_consensus.cc:2747] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:11.653380 32010 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.667677 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:11.666477 31699 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:11.670379 31804 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.674317 31923 tablet_bootstrap.cc:492] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:11.674140 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:11.674767 31923 ts_tablet_manager.cc:1397] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.100s user 0.017s sys 0.011s
W20250831 07:58:11.676558 31773 leader_election.cc:343] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.682760 32014 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:11.691306 31925 tablet_bootstrap.cc:654] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.688062 31730 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:11.700526 32010 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:11.701380 31775 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:11.695286 31923 raft_consensus.cc:357] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.702996 31923 raft_consensus.cc:383] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.703333 31967 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:11.703333 31923 raft_consensus.cc:738] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.703997 31967 raft_consensus.cc:695] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:11.704551 31923 consensus_queue.cc:260] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.704833 31967 consensus_queue.cc:237] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.713258 31924 tablet_bootstrap.cc:492] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:11.713618 31923 ts_tablet_manager.cc:1428] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.038s user 0.008s sys 0.000s
I20250831 07:58:11.713801 31924 ts_tablet_manager.cc:1397] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.255s user 0.007s sys 0.010s
I20250831 07:58:11.715015 31923 tablet_bootstrap.cc:492] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:11.716732 31924 raft_consensus.cc:357] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.717404 31924 raft_consensus.cc:383] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.717664 31924 raft_consensus.cc:738] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.717170 31628 catalog_manager.cc:5582] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:11.718343 31924 consensus_queue.cc:260] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.719172 31628 catalog_manager.cc:5582] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:11.726228 31924 ts_tablet_manager.cc:1428] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.012s user 0.002s sys 0.006s
I20250831 07:58:11.728166 31924 tablet_bootstrap.cc:492] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:11.739517 32004 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.744071 32004 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.755081 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:11.756325 31805 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:11.758417 31699 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:11.759621 32014 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:11.760114 32014 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:11.760540 32014 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.764609 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:11.753564 32004 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:11.766229 31698 leader_election.cc:343] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.766449 31924 tablet_bootstrap.cc:654] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.771413 31925 tablet_bootstrap.cc:492] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:11.771345 31804 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:11.772260 31925 ts_tablet_manager.cc:1397] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.117s user 0.024s sys 0.024s
I20250831 07:58:11.773134 31699 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:11.775115 32004 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:11.775277 31925 raft_consensus.cc:357] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.775797 32004 raft_consensus.cc:695] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:11.776047 31925 raft_consensus.cc:383] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.776357 31925 raft_consensus.cc:738] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.777096 31925 consensus_queue.cc:260] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.777285 32004 consensus_queue.cc:237] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.778548 31630 catalog_manager.cc:5582] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:11.779465 31925 ts_tablet_manager.cc:1428] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 07:58:11.780121 31967 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.780606 31967 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.781975 32014 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.782491 31967 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.783181 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:11.783581 31925 tablet_bootstrap.cc:492] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
W20250831 07:58:11.784467 31775 leader_election.cc:343] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.785254 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:11.785797 31805 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.787071 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:11.790268 31773 leader_election.cc:343] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.790727 31773 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:11.791451 31967 raft_consensus.cc:2747] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:11.792233 31925 tablet_bootstrap.cc:654] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.793300 31923 tablet_bootstrap.cc:654] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.805596 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:58:11.807024 31698 leader_election.cc:343] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:11.800770 32014 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:11.821763 31925 tablet_bootstrap.cc:492] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:11.822312 31925 ts_tablet_manager.cc:1397] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.017s
I20250831 07:58:11.825050 31925 raft_consensus.cc:357] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.825763 31925 raft_consensus.cc:383] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.826059 31925 raft_consensus.cc:738] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.826722 31925 consensus_queue.cc:260] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.825210 31805 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:11.828586 31699 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:11.829502 31925 ts_tablet_manager.cc:1428] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250831 07:58:11.830092 32004 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:11.830377 31925 tablet_bootstrap.cc:492] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:11.830735 32004 raft_consensus.cc:695] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:11.831876 32004 consensus_queue.cc:237] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.840111 31631 catalog_manager.cc:5582] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:11.845283 31882 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.855347 31923 tablet_bootstrap.cc:492] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:11.855872 31923 ts_tablet_manager.cc:1397] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.141s user 0.012s sys 0.015s
I20250831 07:58:11.858632 31923 raft_consensus.cc:357] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.859385 31923 raft_consensus.cc:383] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.859649 31923 raft_consensus.cc:738] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.860345 31923 consensus_queue.cc:260] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.870227 31925 tablet_bootstrap.cc:654] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.874322 31630 catalog_manager.cc:5582] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:11.886049 31923 ts_tablet_manager.cc:1428] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20250831 07:58:11.887423 31923 tablet_bootstrap.cc:492] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:11.891427 31925 tablet_bootstrap.cc:492] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:11.891921 31925 ts_tablet_manager.cc:1397] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.062s user 0.014s sys 0.001s
I20250831 07:58:11.896049 31924 tablet_bootstrap.cc:492] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:11.896620 31924 ts_tablet_manager.cc:1397] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.170s user 0.015s sys 0.054s
I20250831 07:58:11.896519 31925 raft_consensus.cc:357] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.897564 31925 raft_consensus.cc:383] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.897926 31925 raft_consensus.cc:738] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.899447 31925 consensus_queue.cc:260] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.899955 31924 raft_consensus.cc:357] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.900416 31923 tablet_bootstrap.cc:654] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.900617 31924 raft_consensus.cc:383] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.900918 31924 raft_consensus.cc:738] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.901628 31924 consensus_queue.cc:260] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.903590 31924 ts_tablet_manager.cc:1428] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250831 07:58:11.904393 31924 tablet_bootstrap.cc:492] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:11.913689 31925 ts_tablet_manager.cc:1428] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20250831 07:58:11.917641 31883 raft_consensus.cc:1273] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250831 07:58:11.921670 32004 consensus_queue.cc:1035] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:11.924057 31925 tablet_bootstrap.cc:492] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:11.936501 31924 tablet_bootstrap.cc:654] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.939929 31925 tablet_bootstrap.cc:654] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.946626 31805 raft_consensus.cc:1273] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250831 07:58:11.941838 31997 consensus_queue.cc:1035] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:11.942248 31923 tablet_bootstrap.cc:492] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:11.947041 31805 pending_rounds.cc:85] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 0
I20250831 07:58:11.947515 31805 pending_rounds.cc:107] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250831 07:58:11.947566 31923 ts_tablet_manager.cc:1397] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.013s
I20250831 07:58:11.947896 31805 raft_consensus.cc:2887] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:58:11.949815 32014 consensus_queue.cc:1035] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:11.960809 31923 raft_consensus.cc:357] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.961580 31923 raft_consensus.cc:383] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.962044 31923 raft_consensus.cc:738] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.962675 31923 consensus_queue.cc:260] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.964962 31923 ts_tablet_manager.cc:1428] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250831 07:58:11.965930 31923 tablet_bootstrap.cc:492] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:11.973500 31923 tablet_bootstrap.cc:654] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:11.981253 31996 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:11.981778 31996 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.982198 31925 tablet_bootstrap.cc:492] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:11.982784 31925 ts_tablet_manager.cc:1397] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.006s
I20250831 07:58:11.984690 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:11.985469 31733 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 1: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:11.985843 31923 tablet_bootstrap.cc:492] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:11.986266 31923 ts_tablet_manager.cc:1397] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.003s
I20250831 07:58:11.986965 31996 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:11.987243 31999 consensus_queue.cc:1035] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:11.988143 31925 raft_consensus.cc:357] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.988437 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:11.989279 31925 raft_consensus.cc:383] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.989220 31923 raft_consensus.cc:357] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.989642 31925 raft_consensus.cc:738] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.989966 31923 raft_consensus.cc:383] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:11.989815 31848 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:11.990643 31999 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:11.990424 31925 consensus_queue.cc:260] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:11.992853 31925 ts_tablet_manager.cc:1428] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250831 07:58:11.993789 31925 tablet_bootstrap.cc:492] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:11.998482 31999 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:11.990262 31923 raft_consensus.cc:738] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:11.999678 31923 consensus_queue.cc:260] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.001955 31923 ts_tablet_manager.cc:1428] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250831 07:58:12.002089 31996 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.002908 31923 tablet_bootstrap.cc:492] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:12.002580 31996 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.004565 31996 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:12.006618 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:12.006848 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:12.007262 31806 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
W20250831 07:58:12.008198 31849 leader_election.cc:343] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.008848 31848 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:12.009266 31925 tablet_bootstrap.cc:654] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.009359 31923 tablet_bootstrap.cc:654] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.009701 31999 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:12.017712 31999 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:12.022805 31999 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.024762 31954 consensus_queue.cc:1035] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:12.027460 31924 tablet_bootstrap.cc:492] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:12.027958 31924 ts_tablet_manager.cc:1397] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.124s user 0.010s sys 0.010s
I20250831 07:58:12.033777 31924 raft_consensus.cc:357] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.034315 31924 raft_consensus.cc:383] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.034577 31924 raft_consensus.cc:738] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.036296 31999 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.035573 31924 consensus_queue.cc:260] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.047612 31924 ts_tablet_manager.cc:1428] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250831 07:58:12.048666 31924 tablet_bootstrap.cc:492] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:12.050582 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:12.051874 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:12.052443 31999 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:12.053071 31849 leader_election.cc:343] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.052495 31805 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.063800 31924 tablet_bootstrap.cc:654] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.069221 31805 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:12.070469 31848 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:12.071326 31996 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:12.071791 31996 raft_consensus.cc:695] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:12.072659 31996 consensus_queue.cc:237] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.074919 31923 tablet_bootstrap.cc:492] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:12.075347 31923 ts_tablet_manager.cc:1397] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.073s user 0.008s sys 0.020s
I20250831 07:58:12.077800 31923 raft_consensus.cc:357] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.078419 31923 raft_consensus.cc:383] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.079074 31923 raft_consensus.cc:738] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.079784 31923 consensus_queue.cc:260] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.085011 31923 ts_tablet_manager.cc:1428] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20250831 07:58:12.085955 31923 tablet_bootstrap.cc:492] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:12.087292 31924 tablet_bootstrap.cc:492] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:12.087785 31924 ts_tablet_manager.cc:1397] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.039s user 0.016s sys 0.009s
I20250831 07:58:12.056418 31925 tablet_bootstrap.cc:492] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:12.089159 31925 ts_tablet_manager.cc:1397] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.096s user 0.012s sys 0.011s
I20250831 07:58:12.091116 31924 raft_consensus.cc:357] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.091920 31924 raft_consensus.cc:383] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.092199 31924 raft_consensus.cc:738] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.092834 31924 consensus_queue.cc:260] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.094965 31924 ts_tablet_manager.cc:1428] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250831 07:58:12.095908 31924 tablet_bootstrap.cc:492] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:12.103976 31923 tablet_bootstrap.cc:654] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.100142 31967 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.105235 31967 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.107837 31967 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:12.111634 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:12.112428 31730 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:12.113452 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:12.113953 31775 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:12.113664 31630 catalog_manager.cc:5582] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
W20250831 07:58:12.115027 31773 leader_election.cc:343] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.116673 32008 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:12.116860 31924 tablet_bootstrap.cc:654] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.117063 32008 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:12.117439 32008 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.119562 31954 consensus_queue.cc:1035] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:12.124408 32008 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.127136 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:12.127702 31733 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.128664 31923 tablet_bootstrap.cc:492] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:12.129114 31923 ts_tablet_manager.cc:1397] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.004s
I20250831 07:58:12.131529 31923 raft_consensus.cc:357] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.132177 31923 raft_consensus.cc:383] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.132447 31923 raft_consensus.cc:738] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.133067 31923 consensus_queue.cc:260] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.134917 32008 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:12.136096 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:58:12.137518 31773 leader_election.cc:343] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.138976 31923 ts_tablet_manager.cc:1428] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250831 07:58:12.141402 31923 tablet_bootstrap.cc:492] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:12.141958 31733 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:12.142328 31924 tablet_bootstrap.cc:492] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:12.142913 31924 ts_tablet_manager.cc:1397] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.000s
I20250831 07:58:12.143230 31775 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:12.144394 32008 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:12.144869 32008 raft_consensus.cc:695] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:12.145704 32008 consensus_queue.cc:237] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.146083 31925 raft_consensus.cc:357] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.146747 31925 raft_consensus.cc:383] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.146469 31924 raft_consensus.cc:357] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.147001 31925 raft_consensus.cc:738] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.147099 31924 raft_consensus.cc:383] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.147337 31923 tablet_bootstrap.cc:654] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.147606 31925 consensus_queue.cc:260] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.148506 31999 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.149000 31999 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.152259 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:12.156006 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:12.157050 31806 raft_consensus.cc:2391] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 1: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:12.159497 31848 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:12.160764 31996 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.147358 31924 raft_consensus.cc:738] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.165277 31999 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:12.166919 31924 consensus_queue.cc:260] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.167784 32010 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.168268 32010 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.169055 31996 raft_consensus.cc:2747] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:12.168792 31631 catalog_manager.cc:5582] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:12.173677 32010 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:12.175148 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:12.176797 31698 leader_election.cc:343] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.179558 32004 consensus_queue.cc:1035] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:12.182029 31924 ts_tablet_manager.cc:1428] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.039s user 0.007s sys 0.001s
I20250831 07:58:12.182952 31924 tablet_bootstrap.cc:492] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:12.184468 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:12.185573 31806 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:12.187348 31699 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:12.188867 32010 raft_consensus.cc:2802] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:12.193019 32010 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:12.193413 32010 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.197094 32009 consensus_queue.cc:1035] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:12.200603 31967 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.201563 31967 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.205773 31967 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:12.208529 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:12.210654 31775 leader_election.cc:343] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.212311 31925 ts_tablet_manager.cc:1428] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.121s user 0.007s sys 0.000s
I20250831 07:58:12.212812 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:12.213541 31925 tablet_bootstrap.cc:492] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
W20250831 07:58:12.214663 31773 leader_election.cc:343] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.215302 31773 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:12.216832 32009 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:12.218681 31924 tablet_bootstrap.cc:654] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.224697 31967 consensus_queue.cc:1035] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20250831 07:58:12.223574 32010 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.249754 31996 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.250252 31996 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.255445 31996 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:12.264075 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:12.265029 31730 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:12.266255 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:12.266660 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:12.267087 31806 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:12.267491 32010 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:12.267941 31698 leader_election.cc:343] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.272125 31849 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:12.273614 31996 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:12.276206 31996 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:12.276572 31996 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.280442 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:12.281213 31806 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.285610 32004 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.286290 32004 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.288784 32004 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:12.289445 32004 consensus_queue.cc:1035] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:12.292940 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:12.292760 31996 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:12.294624 31698 leader_election.cc:343] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.301872 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:12.336521 31923 tablet_bootstrap.cc:492] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:12.337117 31923 ts_tablet_manager.cc:1397] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.196s user 0.047s sys 0.087s
I20250831 07:58:12.350302 31923 raft_consensus.cc:357] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.351073 31923 raft_consensus.cc:383] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.351344 31923 raft_consensus.cc:738] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.352035 31923 consensus_queue.cc:260] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.362062 31925 tablet_bootstrap.cc:654] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.364638 31923 ts_tablet_manager.cc:1428] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20250831 07:58:12.366094 31923 tablet_bootstrap.cc:492] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:12.374598 32061 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.375118 32061 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.385691 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:12.386427 31730 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.388605 31954 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.389106 31954 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.391036 31954 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:12.394258 31996 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:12.394323 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:12.357642 31805 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:12.400795 32061 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:12.405789 31849 leader_election.cc:343] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:12.414139 32062 log.cc:927] Time spent T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20250831 07:58:12.415175 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:12.416867 31806 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
W20250831 07:58:12.421492 31775 leader_election.cc:343] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.422086 31699 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:12.423451 31699 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:12.424448 32064 raft_consensus.cc:2802] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:12.425043 32064 raft_consensus.cc:695] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:12.423597 32010 raft_consensus.cc:2802] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:12.425920 32010 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:12.426334 32010 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.426146 32064 consensus_queue.cc:237] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.442690 31923 tablet_bootstrap.cc:654] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.443925 31996 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.444615 31996 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:12.445333 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 108379us
I20250831 07:58:12.445672 31924 tablet_bootstrap.cc:492] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:12.446146 31924 ts_tablet_manager.cc:1397] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.263s user 0.015s sys 0.010s
I20250831 07:58:12.447010 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:12.447144 31996 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:12.451406 31680 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 07:58:12.451653 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:12.460146 31883 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.461206 32010 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.461941 31730 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:12.464490 31849 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:12.465613 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:12.466259 31806 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.467286 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:12.468283 31805 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:12.469029 31996 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:12.469514 31996 raft_consensus.cc:695] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:12.470331 32010 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:12.471712 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:12.472309 31880 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.473081 32010 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:12.473596 31848 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:12.475152 31954 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:12.475533 31954 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:12.483112 32004 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250831 07:58:12.478252 31954 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.488592 31924 raft_consensus.cc:357] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.489254 31924 raft_consensus.cc:383] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.469532 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:12.497174 31805 raft_consensus.cc:2391] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 1: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:12.499027 31848 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:12.509140 31849 consensus_peers.cc:489] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 -> Peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Couldn't send request to peer a0d845d01c8c4894895f66e3beae298c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:12.470347 31996 consensus_queue.cc:237] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.517751 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:12.518563 31805 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:12.531991 31773 leader_election.cc:343] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.532644 31773 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:12.533567 32009 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:12.544955 31997 consensus_queue.cc:1035] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:12.489544 31924 raft_consensus.cc:738] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.552500 31924 consensus_queue.cc:260] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.558955 31923 tablet_bootstrap.cc:492] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:12.559624 31923 ts_tablet_manager.cc:1397] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.194s user 0.030s sys 0.048s
I20250831 07:58:12.562768 31806 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:12.563481 31923 raft_consensus.cc:357] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.564436 31923 raft_consensus.cc:383] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.564786 31923 raft_consensus.cc:738] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.565605 31923 consensus_queue.cc:260] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.566744 31880 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:12.568205 31698 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:12.570876 31954 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.583341 31999 raft_consensus.cc:2747] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:12.580559 32010 raft_consensus.cc:2802] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:12.587304 32010 raft_consensus.cc:695] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:12.587738 31924 ts_tablet_manager.cc:1428] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.141s user 0.006s sys 0.002s
I20250831 07:58:12.586810 31925 tablet_bootstrap.cc:492] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:12.588202 32010 consensus_queue.cc:237] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.588922 31925 ts_tablet_manager.cc:1397] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.376s user 0.030s sys 0.049s
I20250831 07:58:12.590303 31923 ts_tablet_manager.cc:1428] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.030s user 0.012s sys 0.005s
I20250831 07:58:12.590464 31630 catalog_manager.cc:5582] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:12.591197 31923 tablet_bootstrap.cc:492] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:12.593423 32067 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.593894 32067 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.597041 31925 raft_consensus.cc:357] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.597689 31925 raft_consensus.cc:383] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.597931 31925 raft_consensus.cc:738] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.598515 31925 consensus_queue.cc:260] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.601503 32067 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:12.602813 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:12.604468 31773 leader_election.cc:343] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.604595 31925 ts_tablet_manager.cc:1428] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250831 07:58:12.606019 31925 tablet_bootstrap.cc:492] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:12.606176 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:12.608183 31775 leader_election.cc:343] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.608652 31775 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:12.601276 31805 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:12.611344 31954 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:12.612524 32009 raft_consensus.cc:2747] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:12.612905 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:12.621980 31924 tablet_bootstrap.cc:492] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:12.619028 32014 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.622910 31804 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.625896 32014 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.644248 32014 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:12.652793 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:12.654345 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:12.656702 31849 leader_election.cc:343] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.658234 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:12.658506 31804 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:12.659530 31848 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:12.660122 31699 leader_election.cc:343] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.660770 31997 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:12.661255 31997 raft_consensus.cc:695] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:12.661820 31924 tablet_bootstrap.cc:654] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
W20250831 07:58:12.662032 31698 leader_election.cc:343] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.662205 31997 consensus_queue.cc:237] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.662424 31698 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:12.663676 32063 raft_consensus.cc:2747] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 07:58:12.667225 31773 consensus_peers.cc:489] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 -> Peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Couldn't send request to peer 8247a38b550649aa9cd6e7668dcaec10. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:12.671973 31925 tablet_bootstrap.cc:654] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.682874 31924 tablet_bootstrap.cc:492] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:12.683378 31924 ts_tablet_manager.cc:1397] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.004s
I20250831 07:58:12.686409 31923 tablet_bootstrap.cc:654] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.688782 31925 tablet_bootstrap.cc:492] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:12.686084 31924 raft_consensus.cc:357] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.689384 31925 ts_tablet_manager.cc:1397] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.084s user 0.012s sys 0.000s
I20250831 07:58:12.689482 31924 raft_consensus.cc:383] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.689924 31924 raft_consensus.cc:738] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.690588 31924 consensus_queue.cc:260] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.690505 31628 catalog_manager.cc:5582] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:12.692003 31925 raft_consensus.cc:357] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.692703 31925 raft_consensus.cc:383] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.692989 31925 raft_consensus.cc:738] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.693614 31925 consensus_queue.cc:260] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.695327 31924 ts_tablet_manager.cc:1428] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.012s user 0.006s sys 0.002s
I20250831 07:58:12.695874 31925 ts_tablet_manager.cc:1428] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250831 07:58:12.696872 31925 tablet_bootstrap.cc:492] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:12.697801 32063 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.698251 32063 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.701810 31924 tablet_bootstrap.cc:492] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:12.702461 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:12.702883 32009 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.703274 31804 raft_consensus.cc:2391] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:12.703325 32009 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.705403 32063 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:12.705940 31925 tablet_bootstrap.cc:654] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.706972 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:12.710711 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:12.712073 31775 leader_election.cc:343] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.705403 32009 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:12.713904 31924 tablet_bootstrap.cc:654] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.713677 31630 catalog_manager.cc:5582] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
W20250831 07:58:12.716027 31773 leader_election.cc:343] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.716472 31773 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:12.717768 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:12.718712 32009 raft_consensus.cc:2747] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:12.719326 31698 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:12.720428 32063 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.727597 31997 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.728145 31997 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.729939 31997 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:12.727588 32009 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250831 07:58:12.733791 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:12.739115 31804 raft_consensus.cc:2466] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:12.743736 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:12.747570 31849 leader_election.cc:343] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.748629 31848 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:12.749627 31997 raft_consensus.cc:2802] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:12.752308 31997 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:12.752784 31997 raft_consensus.cc:3058] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.761391 31923 tablet_bootstrap.cc:492] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:12.762584 31925 tablet_bootstrap.cc:492] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:12.763001 31925 ts_tablet_manager.cc:1397] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.010s
I20250831 07:58:12.763001 31923 ts_tablet_manager.cc:1397] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.172s user 0.015s sys 0.002s
I20250831 07:58:12.750353 31631 catalog_manager.cc:5582] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:12.767943 31924 tablet_bootstrap.cc:492] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:12.767782 31925 raft_consensus.cc:357] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.768430 31924 ts_tablet_manager.cc:1397] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.067s user 0.019s sys 0.000s
I20250831 07:58:12.768590 31925 raft_consensus.cc:383] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.768894 31925 raft_consensus.cc:738] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.769424 32063 raft_consensus.cc:2747] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:12.769484 31925 consensus_queue.cc:260] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.773545 31924 raft_consensus.cc:357] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.775091 31924 raft_consensus.cc:383] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.775482 31924 raft_consensus.cc:738] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.776196 31924 consensus_queue.cc:260] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.774065 31925 ts_tablet_manager.cc:1428] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.011s user 0.002s sys 0.005s
I20250831 07:58:12.778862 31924 ts_tablet_manager.cc:1428] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250831 07:58:12.779654 31925 tablet_bootstrap.cc:492] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:12.780333 31924 tablet_bootstrap.cc:492] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:12.791586 31997 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.793094 31923 raft_consensus.cc:357] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.793799 31923 raft_consensus.cc:383] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.794090 31923 raft_consensus.cc:738] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.794791 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:12.794739 31923 consensus_queue.cc:260] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:12.796087 31849 leader_election.cc:343] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.797477 31923 ts_tablet_manager.cc:1428] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.034s user 0.005s sys 0.001s
I20250831 07:58:12.800076 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:12.800655 31804 raft_consensus.cc:3058] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.801126 31925 tablet_bootstrap.cc:654] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.804572 31923 tablet_bootstrap.cc:492] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:12.805991 31924 tablet_bootstrap.cc:654] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.807478 31997 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:12.810254 31804 raft_consensus.cc:2466] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:12.811376 31848 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:12.813448 31925 tablet_bootstrap.cc:492] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:12.813869 31925 ts_tablet_manager.cc:1397] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.006s
I20250831 07:58:12.817132 31925 raft_consensus.cc:357] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.817955 31925 raft_consensus.cc:383] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.818254 31925 raft_consensus.cc:738] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.818888 31997 raft_consensus.cc:2802] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:12.818835 31925 consensus_queue.cc:260] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.819584 31997 raft_consensus.cc:695] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:12.820035 31923 tablet_bootstrap.cc:654] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.820413 31997 consensus_queue.cc:237] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.830641 31925 ts_tablet_manager.cc:1428] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.016s user 0.002s sys 0.005s
I20250831 07:58:12.832243 31925 tablet_bootstrap.cc:492] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:12.839259 31924 tablet_bootstrap.cc:492] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:12.842018 31924 ts_tablet_manager.cc:1397] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.062s user 0.026s sys 0.007s
I20250831 07:58:12.845059 31923 tablet_bootstrap.cc:492] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:12.845534 31923 ts_tablet_manager.cc:1397] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.009s
I20250831 07:58:12.845534 31628 catalog_manager.cc:5582] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:12.846634 31924 raft_consensus.cc:357] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.847496 31924 raft_consensus.cc:383] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.847791 31924 raft_consensus.cc:738] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.848600 31924 consensus_queue.cc:260] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.850016 31923 raft_consensus.cc:357] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.850656 31923 raft_consensus.cc:383] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.850876 31924 ts_tablet_manager.cc:1428] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20250831 07:58:12.851697 31925 tablet_bootstrap.cc:654] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.852600 31924 tablet_bootstrap.cc:492] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:12.850937 31923 raft_consensus.cc:738] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.856065 31923 consensus_queue.cc:260] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.863577 31923 ts_tablet_manager.cc:1428] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.018s user 0.004s sys 0.003s
I20250831 07:58:12.865113 31923 tablet_bootstrap.cc:492] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:12.873654 31923 tablet_bootstrap.cc:654] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.874712 31924 tablet_bootstrap.cc:654] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.878582 31925 tablet_bootstrap.cc:492] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:12.879181 31925 ts_tablet_manager.cc:1397] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.048s user 0.019s sys 0.008s
I20250831 07:58:12.882196 31925 raft_consensus.cc:357] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.882673 31923 tablet_bootstrap.cc:492] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:12.882936 31925 raft_consensus.cc:383] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.883252 31925 raft_consensus.cc:738] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.883383 31923 ts_tablet_manager.cc:1397] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.000s
I20250831 07:58:12.883947 31924 tablet_bootstrap.cc:492] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:12.884476 31924 ts_tablet_manager.cc:1397] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.006s
I20250831 07:58:12.884008 31925 consensus_queue.cc:260] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.885818 31997 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.886276 31997 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.886576 31923 raft_consensus.cc:357] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.887537 31923 raft_consensus.cc:383] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.887887 31923 raft_consensus.cc:738] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.887866 31924 raft_consensus.cc:357] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.888584 31924 raft_consensus.cc:383] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.888679 31997 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:12.888888 31924 raft_consensus.cc:738] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.888872 31923 consensus_queue.cc:260] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.889722 31924 consensus_queue.cc:260] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.891172 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:12.892302 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:12.893832 31924 ts_tablet_manager.cc:1428] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250831 07:58:12.894707 31923 ts_tablet_manager.cc:1428] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250831 07:58:12.894740 31924 tablet_bootstrap.cc:492] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:12.895197 31925 ts_tablet_manager.cc:1428] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.016s user 0.004s sys 0.004s
I20250831 07:58:12.895931 31923 tablet_bootstrap.cc:492] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:12.900939 31848 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:12.902871 31924 tablet_bootstrap.cc:654] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.902913 31923 tablet_bootstrap.cc:654] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.906152 32078 raft_consensus.cc:2747] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:12.909313 31925 tablet_bootstrap.cc:492] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:12.915585 31924 tablet_bootstrap.cc:492] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:12.916110 31924 ts_tablet_manager.cc:1397] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.003s
I20250831 07:58:12.918234 31924 raft_consensus.cc:357] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.918800 31924 raft_consensus.cc:383] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.919019 31924 raft_consensus.cc:738] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.919739 31923 tablet_bootstrap.cc:492] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:12.919529 31924 consensus_queue.cc:260] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.920258 31923 ts_tablet_manager.cc:1397] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.000s
I20250831 07:58:12.922160 31924 ts_tablet_manager.cc:1428] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250831 07:58:12.924654 31924 tablet_bootstrap.cc:492] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:12.925277 31925 tablet_bootstrap.cc:654] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.926101 31923 raft_consensus.cc:357] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.926759 31923 raft_consensus.cc:383] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.927011 31923 raft_consensus.cc:738] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.927707 31923 consensus_queue.cc:260] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.933209 31923 ts_tablet_manager.cc:1428] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250831 07:58:12.934108 31923 tablet_bootstrap.cc:492] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:12.936558 32067 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.946615 32067 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.952930 31882 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:12.954020 31924 tablet_bootstrap.cc:654] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.952967 32067 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:12.955899 32078 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:12.956626 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:12.958279 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:12.959758 31775 leader_election.cc:343] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:12.960769 31773 leader_election.cc:343] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.961460 31773 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:12.965271 31925 tablet_bootstrap.cc:492] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:12.965456 32063 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:12.965806 31925 ts_tablet_manager.cc:1397] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.057s user 0.012s sys 0.009s
I20250831 07:58:12.965600 32065 raft_consensus.cc:2747] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:12.966485 32065 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:12.967031 32065 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.968851 31923 tablet_bootstrap.cc:654] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:12.976488 31925 raft_consensus.cc:357] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.977310 31925 raft_consensus.cc:383] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:12.977636 31925 raft_consensus.cc:738] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:12.978366 31925 consensus_queue.cc:260] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:12.982901 32065 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:12.985543 31925 ts_tablet_manager.cc:1428] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.019s user 0.008s sys 0.002s
I20250831 07:58:12.986423 31925 tablet_bootstrap.cc:492] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:12.988868 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:12.991089 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:12.992952 31775 leader_election.cc:343] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:12.995055 31773 leader_election.cc:343] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:12.995651 31773 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:12.997076 32065 raft_consensus.cc:2747] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:13.003371 31924 tablet_bootstrap.cc:492] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:13.004050 31924 ts_tablet_manager.cc:1397] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.080s user 0.014s sys 0.012s
I20250831 07:58:13.008065 31924 raft_consensus.cc:357] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.012686 31924 raft_consensus.cc:383] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.013064 31924 raft_consensus.cc:738] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.014037 31924 consensus_queue.cc:260] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.017560 31924 ts_tablet_manager.cc:1428] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250831 07:58:13.018548 31924 tablet_bootstrap.cc:492] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:13.021512 31923 tablet_bootstrap.cc:492] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:13.022009 31923 ts_tablet_manager.cc:1397] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.088s user 0.024s sys 0.019s
I20250831 07:58:13.024868 31924 tablet_bootstrap.cc:654] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.024717 31923 raft_consensus.cc:357] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.025259 32063 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.025403 31923 raft_consensus.cc:383] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.026079 31923 raft_consensus.cc:738] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.026150 32063 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.026856 31923 consensus_queue.cc:260] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.027804 31954 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.034973 32078 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.035414 31997 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:13.033954 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.035521 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:13.033087 31954 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.032824 32063 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.038295 31954 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:13.033943 31923 ts_tablet_manager.cc:1428] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20250831 07:58:13.040746 31923 tablet_bootstrap.cc:492] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:13.045235 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:13.035532 32078 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.046136 31730 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:13.049820 31923 tablet_bootstrap.cc:654] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.054769 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.055630 31808 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:13.063050 31848 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:13.064524 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.069633 31808 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:13.067520 31699 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:13.049165 32078 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:13.071969 32010 raft_consensus.cc:2747] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:13.076414 32065 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.076941 32065 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.078898 32078 raft_consensus.cc:2802] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:13.084497 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:13.092267 32078 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:13.086874 32065 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.088109 32064 consensus_queue.cc:1035] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:13.085848 32010 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:13.093011 31775 leader_election.cc:343] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.093420 31848 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:13.095837 32078 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.098176 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:13.108253 31730 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:13.095777 32010 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.103780 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:13.104084 31954 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:13.110018 31954 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:13.110508 31954 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.110846 32010 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.108829 31925 tablet_bootstrap.cc:654] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.114485 32014 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:13.113910 31773 leader_election.cc:343] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.115106 32014 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.115366 31773 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:13.118096 32014 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.121332 32067 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:13.128274 31882 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.128638 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:13.128316 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:13.125552 32064 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
W20250831 07:58:13.140024 31698 leader_election.cc:343] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.144577 31924 tablet_bootstrap.cc:492] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:13.145236 31924 ts_tablet_manager.cc:1397] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.127s user 0.023s sys 0.053s
I20250831 07:58:13.149298 31924 raft_consensus.cc:357] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.150168 31924 raft_consensus.cc:383] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.150457 31924 raft_consensus.cc:738] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.151257 31924 consensus_queue.cc:260] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.151989 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.152870 31808 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:13.154263 31924 ts_tablet_manager.cc:1428] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250831 07:58:13.155542 31924 tablet_bootstrap.cc:492] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:13.158771 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.175287 31730 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.172483 31808 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:13.178884 32064 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.181064 32064 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.182962 32064 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.186182 31923 tablet_bootstrap.cc:492] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:13.186657 31923 ts_tablet_manager.cc:1397] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.146s user 0.041s sys 0.057s
I20250831 07:58:13.187470 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:13.189183 31923 raft_consensus.cc:357] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.189857 31923 raft_consensus.cc:383] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.190152 31923 raft_consensus.cc:738] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
W20250831 07:58:13.188943 31698 leader_election.cc:343] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.192520 31923 consensus_queue.cc:260] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.196758 31996 consensus_queue.cc:1035] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20250831 07:58:13.211861 32078 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.213138 31699 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:13.215554 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.228854 32064 raft_consensus.cc:2802] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:13.229408 32014 consensus_queue.cc:1035] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:13.221760 31924 tablet_bootstrap.cc:654] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.223755 31923 ts_tablet_manager.cc:1428] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.037s user 0.007s sys 0.015s
I20250831 07:58:13.229533 31699 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:13.218987 31954 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.231002 31923 tablet_bootstrap.cc:492] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:13.230166 32064 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:13.235975 32014 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:13.238154 31883 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:13.238983 32014 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:13.239351 32014 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.238618 32064 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:13.249413 31699 leader_election.cc:343] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.251318 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:13.256466 31805 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.249537 32078 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:13.262712 31954 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:13.265183 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:13.266130 31806 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.255713 31699 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:13.270869 31805 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:13.273587 31806 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:13.274528 32010 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:13.274441 31848 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:13.276170 31848 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:13.279446 31954 raft_consensus.cc:2802] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:13.282495 31996 consensus_queue.cc:1035] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:13.280535 31954 raft_consensus.cc:695] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:13.285799 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:13.286509 31733 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.281145 32108 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:13.285836 31954 consensus_queue.cc:237] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.288507 32108 raft_consensus.cc:695] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:13.302681 32108 consensus_queue.cc:237] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.300504 31925 tablet_bootstrap.cc:492] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:13.304718 31925 ts_tablet_manager.cc:1397] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.318s user 0.004s sys 0.024s
I20250831 07:58:13.292412 32013 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.295508 31733 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:13.306535 32013 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.309810 31730 raft_consensus.cc:3058] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.310528 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:13.298138 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:13.307792 31925 raft_consensus.cc:357] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.313969 31925 raft_consensus.cc:383] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.313243 32013 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.317992 32064 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.314216 31925 raft_consensus.cc:738] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.321821 32067 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.322832 32067 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.322708 31925 consensus_queue.cc:260] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.326120 32067 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.327165 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:13.327576 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:13.327811 31883 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:13.328989 31773 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:58:13.329517 31775 leader_election.cc:343] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.329988 32068 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:13.330374 32068 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:13.330889 32068 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.332907 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.334214 32078 consensus_queue.cc:1035] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:13.335096 31996 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.333966 31808 raft_consensus.cc:2391] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:13.335713 31996 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.337393 31699 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:13.300830 32014 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.362988 31925 ts_tablet_manager.cc:1428] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.058s user 0.008s sys 0.001s
I20250831 07:58:13.364428 31925 tablet_bootstrap.cc:492] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:13.351274 32013 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:13.353859 31996 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:13.371603 31924 tablet_bootstrap.cc:492] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:13.372900 31924 ts_tablet_manager.cc:1397] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.218s user 0.028s sys 0.025s
I20250831 07:58:13.377926 31923 tablet_bootstrap.cc:654] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.375989 31732 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 1: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:13.386102 32064 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.386241 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:13.384119 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.388003 31808 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:13.388623 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:13.387028 31728 raft_consensus.cc:2391] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 1: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:13.389199 31805 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.390725 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:13.391624 31883 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.396236 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:13.397059 31804 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:13.405264 31805 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:13.405094 31631 catalog_manager.cc:5582] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:13.413720 31924 raft_consensus.cc:357] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.414225 31924 raft_consensus.cc:383] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.414436 31924 raft_consensus.cc:738] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.414994 31924 consensus_queue.cc:260] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.413367 31848 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:13.416824 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:13.413120 32014 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.421761 31698 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:13.421957 31924 ts_tablet_manager.cc:1428] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.049s user 0.004s sys 0.001s
I20250831 07:58:13.423055 31924 tablet_bootstrap.cc:492] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:13.424546 31997 consensus_queue.cc:1035] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:13.425884 31699 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:13.426898 32013 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250831 07:58:13.433532 31925 tablet_bootstrap.cc:654] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.432798 32014 raft_consensus.cc:2802] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:13.444819 31883 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:13.458405 31954 raft_consensus.cc:2762] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250831 07:58:13.432576 32068 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.461256 32068 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.449429 32014 raft_consensus.cc:695] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:13.465976 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:13.466527 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:13.466261 32014 consensus_queue.cc:237] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.467116 31883 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:13.467307 31775 leader_election.cc:343] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.483284 31923 tablet_bootstrap.cc:492] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:13.483762 31923 ts_tablet_manager.cc:1397] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.253s user 0.005s sys 0.016s
I20250831 07:58:13.491696 32013 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.492226 32013 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.492498 31925 tablet_bootstrap.cc:492] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:13.493104 31925 ts_tablet_manager.cc:1397] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.129s user 0.011s sys 0.007s
I20250831 07:58:13.493743 32013 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.495102 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.495340 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:13.495970 31806 raft_consensus.cc:2466] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
W20250831 07:58:13.496455 31698 leader_election.cc:343] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.497291 31699 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:13.498023 32014 raft_consensus.cc:2802] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:13.499670 31925 raft_consensus.cc:357] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.500422 31924 tablet_bootstrap.cc:654] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.505282 31883 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:13.500809 31923 raft_consensus.cc:357] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.507171 31923 raft_consensus.cc:383] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.506507 31773 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:13.508698 32068 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:13.509267 32068 raft_consensus.cc:695] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:13.510054 32068 consensus_queue.cc:237] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.507699 31923 raft_consensus.cc:738] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.512697 31923 consensus_queue.cc:260] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.513523 31925 raft_consensus.cc:383] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.513825 31925 raft_consensus.cc:738] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.514460 31925 consensus_queue.cc:260] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.514372 31630 catalog_manager.cc:5582] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:13.503980 32014 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:13.522912 31954 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.521734 32014 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.523959 31954 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.528545 31923 ts_tablet_manager.cc:1428] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.044s user 0.004s sys 0.002s
I20250831 07:58:13.529898 31923 tablet_bootstrap.cc:492] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:13.546895 31925 ts_tablet_manager.cc:1428] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.053s user 0.009s sys 0.002s
I20250831 07:58:13.547978 31925 tablet_bootstrap.cc:492] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:13.557701 32014 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.578343 32014 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.580406 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:13.586364 31804 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.586728 31924 tablet_bootstrap.cc:492] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:13.587185 31924 ts_tablet_manager.cc:1397] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.164s user 0.016s sys 0.002s
I20250831 07:58:13.582082 31925 tablet_bootstrap.cc:654] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.585620 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:58:13.589586 31698 leader_election.cc:343] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:13.592849 31680 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 07:58:13.593362 31924 raft_consensus.cc:357] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.594064 31924 raft_consensus.cc:383] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.594332 31924 raft_consensus.cc:738] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.594946 31924 consensus_queue.cc:260] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.601446 31804 raft_consensus.cc:2466] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:13.602820 31954 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:13.604817 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.605813 31806 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:13.610498 31924 ts_tablet_manager.cc:1428] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.023s user 0.001s sys 0.006s
I20250831 07:58:13.625705 31924 tablet_bootstrap.cc:492] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:13.606101 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:13.627200 31848 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:13.628634 31954 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:13.628994 31954 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:13.629314 31954 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.629554 31730 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:13.633169 31925 tablet_bootstrap.cc:492] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:13.633714 31925 ts_tablet_manager.cc:1397] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.086s user 0.016s sys 0.009s
I20250831 07:58:13.635888 31923 tablet_bootstrap.cc:654] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.636961 31925 raft_consensus.cc:357] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.637727 31925 raft_consensus.cc:383] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.638054 31925 raft_consensus.cc:738] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.607416 31699 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:13.639070 32013 raft_consensus.cc:2802] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:13.639971 32013 raft_consensus.cc:695] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:13.640623 31925 consensus_queue.cc:260] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.643676 31925 ts_tablet_manager.cc:1428] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250831 07:58:13.643886 31924 tablet_bootstrap.cc:654] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.640661 32013 consensus_queue.cc:237] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.644809 31925 tablet_bootstrap.cc:492] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:13.671137 31954 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.680867 31925 tablet_bootstrap.cc:654] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.683387 31924 tablet_bootstrap.cc:492] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:13.683861 31924 ts_tablet_manager.cc:1397] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.021s
I20250831 07:58:13.688607 31924 raft_consensus.cc:357] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.689101 31923 tablet_bootstrap.cc:492] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:13.689288 31924 raft_consensus.cc:383] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.689570 31923 ts_tablet_manager.cc:1397] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.160s user 0.011s sys 0.013s
I20250831 07:58:13.689579 31924 raft_consensus.cc:738] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.690443 31924 consensus_queue.cc:260] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.691335 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:13.691951 31730 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.692572 31924 ts_tablet_manager.cc:1428] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250831 07:58:13.695109 31924 tablet_bootstrap.cc:492] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:13.692394 31923 raft_consensus.cc:357] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.690866 31628 catalog_manager.cc:5582] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:13.700484 31923 raft_consensus.cc:383] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.700766 31924 tablet_bootstrap.cc:654] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.700807 31923 raft_consensus.cc:738] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.704046 31630 catalog_manager.cc:5582] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:13.707331 31923 consensus_queue.cc:260] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.709005 31923 ts_tablet_manager.cc:1428] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250831 07:58:13.709400 32067 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.709846 31923 tablet_bootstrap.cc:492] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:13.712167 32013 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.713541 31924 tablet_bootstrap.cc:492] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:13.712644 32013 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.714704 31924 ts_tablet_manager.cc:1397] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.000s
I20250831 07:58:13.715956 31923 tablet_bootstrap.cc:654] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.717471 31924 raft_consensus.cc:357] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.718195 31924 raft_consensus.cc:383] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.718529 31924 raft_consensus.cc:738] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.719175 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.719483 32013 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.710376 32067 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.719201 31924 consensus_queue.cc:260] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.720033 31806 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:13.721916 31699 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:13.722142 31924 ts_tablet_manager.cc:1428] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250831 07:58:13.722468 31923 tablet_bootstrap.cc:492] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:13.723071 32064 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:13.723363 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:13.722800 32067 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.723446 32064 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:13.719172 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:13.724197 32064 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.723362 31923 ts_tablet_manager.cc:1397] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20250831 07:58:13.725116 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:13.725732 31732 raft_consensus.cc:2466] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:13.727222 31923 raft_consensus.cc:357] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.726920 31775 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:13.729401 31923 raft_consensus.cc:383] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250831 07:58:13.729529 31773 leader_election.cc:343] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.729748 31923 raft_consensus.cc:738] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.728104 31924 tablet_bootstrap.cc:492] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
W20250831 07:58:13.730593 31698 leader_election.cc:343] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.732170 32065 raft_consensus.cc:2802] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:13.732522 32065 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:13.732865 32065 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.738603 31997 consensus_queue.cc:1035] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:13.729064 31925 tablet_bootstrap.cc:492] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:13.747321 31925 ts_tablet_manager.cc:1397] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.103s user 0.014s sys 0.012s
I20250831 07:58:13.746986 32065 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.750134 31924 tablet_bootstrap.cc:654] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.730471 31923 consensus_queue.cc:260] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.750456 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:13.751053 31733 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.752832 31923 ts_tablet_manager.cc:1428] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.028s user 0.003s sys 0.002s
I20250831 07:58:13.749928 31925 raft_consensus.cc:357] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.753546 31925 raft_consensus.cc:383] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.753894 31925 raft_consensus.cc:738] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.753775 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:13.754765 31925 consensus_queue.cc:260] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:13.755630 31773 leader_election.cc:343] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.757135 31925 ts_tablet_manager.cc:1428] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250831 07:58:13.758054 31923 tablet_bootstrap.cc:492] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:13.760823 32065 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.762394 31925 tablet_bootstrap.cc:492] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:13.766531 32065 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.767160 32065 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.770139 31923 tablet_bootstrap.cc:654] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.774719 31730 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:13.774993 32064 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.777644 31924 tablet_bootstrap.cc:492] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:13.778821 31925 tablet_bootstrap.cc:654] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.779340 31924 ts_tablet_manager.cc:1397] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.056s user 0.014s sys 0.003s
I20250831 07:58:13.780697 31954 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:13.784379 32064 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.786180 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:13.786877 32067 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.787318 32067 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:13.787575 31698 leader_election.cc:343] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.789178 32067 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.789537 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:13.789731 32064 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.790117 31805 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.790390 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:13.790180 32064 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.790932 31804 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.792129 32064 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.791802 31631 catalog_manager.cc:5582] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:13.775328 31997 consensus_queue.cc:1035] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250831 07:58:13.794615 32065 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.795744 31733 raft_consensus.cc:2466] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:13.799149 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:13.800088 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:13.800312 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:13.801759 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:13.802088 31775 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:13.802349 31882 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:13.803218 32065 raft_consensus.cc:2802] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:13.803891 32065 raft_consensus.cc:695] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:13.804718 32065 consensus_queue.cc:237] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.805946 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.806535 31806 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:13.808079 31699 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:58:13.808462 31775 leader_election.cc:343] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:13.808869 31698 leader_election.cc:343] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.794008 31849 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:58:13.809332 31773 leader_election.cc:343] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.809896 32064 raft_consensus.cc:2802] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:13.810350 32078 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:13.810436 32064 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:13.810442 31773 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:13.811098 32064 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.811033 32078 raft_consensus.cc:695] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:13.811676 32067 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:13.812238 31804 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:13.794929 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:13.812093 32078 consensus_queue.cc:237] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.812088 32067 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:13.814003 32067 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.821949 31996 consensus_queue.cc:1035] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:13.835871 31923 tablet_bootstrap.cc:492] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:13.836388 31923 ts_tablet_manager.cc:1397] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.079s user 0.010s sys 0.003s
I20250831 07:58:13.813357 31805 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:13.842130 31923 raft_consensus.cc:357] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.842917 31923 raft_consensus.cc:383] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.843228 31923 raft_consensus.cc:738] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.852663 31923 consensus_queue.cc:260] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.858842 32064 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.862748 31923 ts_tablet_manager.cc:1428] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.026s user 0.002s sys 0.005s
I20250831 07:58:13.865618 31924 raft_consensus.cc:357] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.867472 31924 raft_consensus.cc:383] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.867837 31924 raft_consensus.cc:738] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.866485 32067 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.868747 31924 consensus_queue.cc:260] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.877885 31699 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:13.881067 31924 ts_tablet_manager.cc:1428] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.101s user 0.007s sys 0.001s
I20250831 07:58:13.881331 31923 tablet_bootstrap.cc:492] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:13.881989 31924 tablet_bootstrap.cc:492] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:13.883092 32014 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:13.887579 31925 tablet_bootstrap.cc:492] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:13.883589 32014 raft_consensus.cc:695] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:13.888780 31925 ts_tablet_manager.cc:1397] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.127s user 0.013s sys 0.003s
I20250831 07:58:13.889806 31628 catalog_manager.cc:5582] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:13.886044 31630 catalog_manager.cc:5582] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:13.893872 31730 raft_consensus.cc:2391] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 1: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:13.888685 32014 consensus_queue.cc:237] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.898334 32064 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.899617 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:13.902006 32067 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.904829 32013 consensus_queue.cc:1035] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:13.906838 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:58:13.907152 31698 leader_election.cc:343] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.907647 31882 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.908264 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:13.910511 31775 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:13.912410 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:13.913106 31805 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:13.925393 32078 consensus_queue.cc:1035] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:13.932353 31923 tablet_bootstrap.cc:654] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.933610 31925 raft_consensus.cc:357] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.936488 31925 raft_consensus.cc:383] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:13.936695 31925 raft_consensus.cc:738] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:13.937196 31925 consensus_queue.cc:260] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:13.948143 31775 leader_election.cc:343] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:13.954815 31925 ts_tablet_manager.cc:1428] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.066s user 0.005s sys 0.000s
I20250831 07:58:13.955631 31925 tablet_bootstrap.cc:492] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:13.966082 32078 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.968050 32078 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.968469 31882 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:13.969978 31773 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:13.971904 31996 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.972529 31996 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.976567 31924 tablet_bootstrap.cc:654] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:13.977501 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.978358 31804 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:13.979573 32014 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.983860 31848 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:13.986521 31954 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:13.986912 31954 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:13.987473 31996 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:13.969995 32078 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:13.989550 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:13.976362 32065 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:13.989848 32014 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:13.990841 32065 raft_consensus.cc:695] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:13.987977 31954 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.001966 32014 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:13.996134 32065 consensus_queue.cc:237] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.017221 32013 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:13.997262 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:14.003875 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:13.976919 31630 catalog_manager.cc:5582] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:14.005628 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:14.010952 31925 tablet_bootstrap.cc:654] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.014899 32064 consensus_queue.cc:1035] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:14.023577 31804 raft_consensus.cc:2391] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 1: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:14.017838 32013 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.026822 31848 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:14.029544 32078 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.035352 31805 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:14.035903 32068 raft_consensus.cc:2747] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:14.024317 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:14.037566 31732 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:14.042380 32013 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.024633 31808 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:14.047128 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:14.047977 31699 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:14.044132 31954 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.050959 32014 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:14.051324 32014 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:14.051674 32014 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.052765 31882 raft_consensus.cc:2391] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:14.053829 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:14.054461 31805 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.047006 32146 consensus_queue.cc:1035] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:14.049396 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:14.055644 31954 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:14.064088 31699 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:14.068804 32064 raft_consensus.cc:2802] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:14.069272 32064 raft_consensus.cc:695] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:58:14.058509 31698 leader_election.cc:343] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.070091 32064 consensus_queue.cc:237] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.073515 31628 catalog_manager.cc:5582] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
W20250831 07:58:14.081517 31775 consensus_peers.cc:489] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 -> Peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Couldn't send request to peer a0d845d01c8c4894895f66e3beae298c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:14.083562 32014 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.089398 31923 tablet_bootstrap.cc:492] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:14.089955 31923 ts_tablet_manager.cc:1397] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.209s user 0.020s sys 0.044s
I20250831 07:58:14.099578 31924 tablet_bootstrap.cc:492] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:14.100279 31924 ts_tablet_manager.cc:1397] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.218s user 0.016s sys 0.011s
I20250831 07:58:14.108400 31924 raft_consensus.cc:357] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.109119 31924 raft_consensus.cc:383] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.109396 31924 raft_consensus.cc:738] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.109964 31924 consensus_queue.cc:260] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.114845 32078 raft_consensus.cc:2747] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:14.117309 31924 ts_tablet_manager.cc:1428] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20250831 07:58:14.118614 31924 tablet_bootstrap.cc:492] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
W20250831 07:58:14.120245 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 58915us
I20250831 07:58:14.121891 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:14.123262 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:14.116469 32014 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.116401 31805 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:14.124228 31730 raft_consensus.cc:2391] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 1: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:14.128830 31848 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:14.129668 31954 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:14.130229 31954 raft_consensus.cc:695] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:14.131130 31954 consensus_queue.cc:237] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.137306 31925 tablet_bootstrap.cc:492] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:14.137817 31925 ts_tablet_manager.cc:1397] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.182s user 0.019s sys 0.027s
I20250831 07:58:14.138381 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:14.145851 31925 raft_consensus.cc:357] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.146567 31925 raft_consensus.cc:383] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.146834 31925 raft_consensus.cc:738] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.147431 31925 consensus_queue.cc:260] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.149606 31925 ts_tablet_manager.cc:1428] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
W20250831 07:58:14.149668 31699 leader_election.cc:343] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.150171 31699 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:14.150628 31923 raft_consensus.cc:357] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.151302 31923 raft_consensus.cc:383] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.151556 31923 raft_consensus.cc:738] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.151961 32152 raft_consensus.cc:2747] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:14.152180 31923 consensus_queue.cc:260] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.154318 31923 ts_tablet_manager.cc:1428] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.064s user 0.004s sys 0.000s
I20250831 07:58:14.159909 31628 catalog_manager.cc:5582] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:14.164121 31925 tablet_bootstrap.cc:492] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:14.172235 31882 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.198216 31698 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:14.201440 31923 tablet_bootstrap.cc:492] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:14.213193 32064 raft_consensus.cc:2747] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250831 07:58:14.226033 31925 tablet_bootstrap.cc:654] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.229720 31924 tablet_bootstrap.cc:654] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.233337 31997 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:14.233839 31997 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.239144 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:14.239378 31997 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:14.241827 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:14.242455 31806 raft_consensus.cc:2466] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:14.244181 32064 consensus_queue.cc:1035] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:14.244457 31628 catalog_manager.cc:5582] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:14.247255 31848 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:14.248960 31923 tablet_bootstrap.cc:654] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.254437 32078 raft_consensus.cc:2802] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:14.256346 32078 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:14.256754 32078 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:14.258234 31925 tablet_bootstrap.cc:492] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:14.258754 31925 ts_tablet_manager.cc:1397] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.008s
I20250831 07:58:14.263397 32078 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.267935 32013 consensus_queue.cc:1035] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:14.268489 31924 tablet_bootstrap.cc:492] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:14.268920 31924 ts_tablet_manager.cc:1397] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.151s user 0.012s sys 0.013s
I20250831 07:58:14.270835 31925 raft_consensus.cc:357] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.271477 31925 raft_consensus.cc:383] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.272053 31925 raft_consensus.cc:738] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.272680 31925 consensus_queue.cc:260] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.275210 31880 raft_consensus.cc:1238] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:14.275501 31925 ts_tablet_manager.cc:1428] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.016s user 0.004s sys 0.003s
I20250831 07:58:14.277088 31925 tablet_bootstrap.cc:492] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:14.280164 32013 consensus_queue.cc:1046] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:14.281033 32064 raft_consensus.cc:3053] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:14.281385 32064 raft_consensus.cc:738] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:14.282219 32064 consensus_queue.cc:260] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.283393 32064 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:14.283725 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:14.271512 31924 raft_consensus.cc:357] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.284459 31924 raft_consensus.cc:383] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.284716 31924 raft_consensus.cc:738] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.285432 32078 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:14.285328 31924 consensus_queue.cc:260] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.288988 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:14.293164 32013 consensus_queue.cc:1035] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:14.304739 31924 ts_tablet_manager.cc:1428] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20250831 07:58:14.305904 31924 tablet_bootstrap.cc:492] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:14.308476 32078 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:14.309479 31732 raft_consensus.cc:2466] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:14.310976 31849 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:14.312255 31997 raft_consensus.cc:2802] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:14.312712 31997 raft_consensus.cc:695] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:14.313472 31997 consensus_queue.cc:237] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:14.316620 31773 consensus_peers.cc:489] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 -> Peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Couldn't send request to peer 8247a38b550649aa9cd6e7668dcaec10. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:14.317139 32065 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:14.317588 32065 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.320330 31925 tablet_bootstrap.cc:654] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.325124 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:14.326224 31924 tablet_bootstrap.cc:654] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.326150 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:14.326350 31773 leader_election.cc:343] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.327412 32078 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 07:58:14.330818 31775 leader_election.cc:343] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.331287 31775 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:14.332020 32067 raft_consensus.cc:2747] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:14.319551 32065 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.337661 32013 consensus_queue.cc:1035] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:14.345472 32067 consensus_queue.cc:1035] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:14.345678 31923 tablet_bootstrap.cc:492] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:14.349329 31923 ts_tablet_manager.cc:1397] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.148s user 0.021s sys 0.008s
I20250831 07:58:14.374641 31923 raft_consensus.cc:357] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.375352 31923 raft_consensus.cc:383] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.375569 32067 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:14.375620 31923 raft_consensus.cc:738] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.378659 31923 consensus_queue.cc:260] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.381737 31925 tablet_bootstrap.cc:492] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:14.382375 31925 ts_tablet_manager.cc:1397] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.105s user 0.011s sys 0.009s
I20250831 07:58:14.376127 32067 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.383703 31628 catalog_manager.cc:5582] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 2, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 2 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:14.390764 32064 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:14.391767 32067 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.391496 32064 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.394037 32064 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.395948 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:14.400962 31923 ts_tablet_manager.cc:1428] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.051s user 0.004s sys 0.003s
I20250831 07:58:14.401865 31923 tablet_bootstrap.cc:492] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:14.408782 31924 tablet_bootstrap.cc:492] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:14.408969 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:14.409397 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:14.409502 31924 ts_tablet_manager.cc:1397] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.104s user 0.014s sys 0.020s
I20250831 07:58:14.410701 32067 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:14.410859 31773 leader_election.cc:343] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.411170 32067 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.413857 31925 raft_consensus.cc:357] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.414525 31925 raft_consensus.cc:383] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.414768 31925 raft_consensus.cc:738] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.415305 31925 consensus_queue.cc:260] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.418609 31925 ts_tablet_manager.cc:1428] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
W20250831 07:58:14.418859 31775 leader_election.cc:343] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.418996 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:14.419260 31775 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:14.419886 32068 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 07:58:14.420105 31773 leader_election.cc:343] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.420105 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:14.420907 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:14.421392 31883 raft_consensus.cc:2391] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:14.422049 31925 tablet_bootstrap.cc:492] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:14.422282 31728 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.426290 31698 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:14.431366 32067 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.442415 32151 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.442617 31923 tablet_bootstrap.cc:654] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.455418 32065 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 07:58:14.462561 31775 leader_election.cc:343] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.463034 31775 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:14.464082 32146 raft_consensus.cc:2747] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:14.466591 31924 raft_consensus.cc:357] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.467283 31924 raft_consensus.cc:383] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.467556 31924 raft_consensus.cc:738] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.468202 31924 consensus_queue.cc:260] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.470319 31924 ts_tablet_manager.cc:1428] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.060s user 0.004s sys 0.000s
I20250831 07:58:14.473433 31924 tablet_bootstrap.cc:492] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:14.482188 31925 tablet_bootstrap.cc:654] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.485044 32151 raft_consensus.cc:2747] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:14.496717 31923 tablet_bootstrap.cc:492] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:14.497311 31923 ts_tablet_manager.cc:1397] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.096s user 0.007s sys 0.007s
I20250831 07:58:14.499619 31924 tablet_bootstrap.cc:654] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.502254 31923 raft_consensus.cc:357] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.504220 31923 raft_consensus.cc:383] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.504572 31923 raft_consensus.cc:738] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.505246 31923 consensus_queue.cc:260] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.512408 32068 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:14.513300 32078 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:14.513793 32078 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.515367 31923 ts_tablet_manager.cc:1428] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.018s user 0.004s sys 0.003s
I20250831 07:58:14.516292 31923 tablet_bootstrap.cc:492] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:14.517632 32078 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:14.517994 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:14.520339 31849 leader_election.cc:343] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.520780 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:14.521406 31807 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:14.522670 31848 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:14.523586 32078 raft_consensus.cc:2802] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:14.524019 32078 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:14.524381 32078 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.531291 31925 tablet_bootstrap.cc:492] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:14.531877 31925 ts_tablet_manager.cc:1397] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.110s user 0.005s sys 0.014s
I20250831 07:58:14.541846 32078 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.543309 31923 tablet_bootstrap.cc:654] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.545334 31924 tablet_bootstrap.cc:492] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:14.545810 31924 ts_tablet_manager.cc:1397] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.002s
I20250831 07:58:14.549383 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:14.551409 31924 raft_consensus.cc:357] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:14.551769 31849 leader_election.cc:343] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.549383 31925 raft_consensus.cc:357] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.553160 31925 raft_consensus.cc:383] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.553452 31925 raft_consensus.cc:738] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.554045 31925 consensus_queue.cc:260] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.552121 31924 raft_consensus.cc:383] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.554994 31924 raft_consensus.cc:738] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.555586 31924 consensus_queue.cc:260] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.556046 31923 tablet_bootstrap.cc:492] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:14.555984 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:14.556553 31923 ts_tablet_manager.cc:1397] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.002s
I20250831 07:58:14.557708 31924 ts_tablet_manager.cc:1428] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250831 07:58:14.558581 31924 tablet_bootstrap.cc:492] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:14.559536 31923 raft_consensus.cc:357] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.560393 31923 raft_consensus.cc:383] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.560663 31923 raft_consensus.cc:738] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.561321 31923 consensus_queue.cc:260] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.556582 31807 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.563515 31923 ts_tablet_manager.cc:1428] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20250831 07:58:14.567869 31923 tablet_bootstrap.cc:492] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:14.572901 32151 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:14.574709 31923 tablet_bootstrap.cc:654] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.574990 31925 ts_tablet_manager.cc:1428] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
I20250831 07:58:14.584602 31925 tablet_bootstrap.cc:492] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:14.590829 31925 tablet_bootstrap.cc:654] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.594178 31924 tablet_bootstrap.cc:654] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.599776 31883 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.603045 31807 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:14.604859 31848 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:14.605556 31997 raft_consensus.cc:2802] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:14.606051 31997 raft_consensus.cc:695] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:14.606827 31997 consensus_queue.cc:237] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.607909 32078 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:14.612512 32064 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:14.614069 31925 tablet_bootstrap.cc:492] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:14.614503 31925 ts_tablet_manager.cc:1397] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.008s
I20250831 07:58:14.621109 31954 consensus_queue.cc:1035] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:14.620610 31925 raft_consensus.cc:357] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.622828 31925 raft_consensus.cc:383] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.623128 31925 raft_consensus.cc:738] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.623838 31925 consensus_queue.cc:260] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.625018 31924 tablet_bootstrap.cc:492] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:14.625497 31924 ts_tablet_manager.cc:1397] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.004s
I20250831 07:58:14.632387 31923 tablet_bootstrap.cc:492] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:14.632987 31923 ts_tablet_manager.cc:1397] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.003s
I20250831 07:58:14.641439 32014 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:14.641939 32014 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.643783 32014 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.645749 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:14.646339 31806 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:14.680934 31925 ts_tablet_manager.cc:1428] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.066s user 0.008s sys 0.002s
I20250831 07:58:14.682014 31699 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:14.684648 32064 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:14.685034 32064 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:14.685372 32064 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.685534 31925 tablet_bootstrap.cc:492] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:14.691943 31631 catalog_manager.cc:5582] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:14.693557 31923 raft_consensus.cc:357] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.694240 31923 raft_consensus.cc:383] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.694501 31923 raft_consensus.cc:738] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.691967 31924 raft_consensus.cc:357] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.696779 31924 raft_consensus.cc:383] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.697049 31924 raft_consensus.cc:738] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.697631 31924 consensus_queue.cc:260] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.698921 31923 consensus_queue.cc:260] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.701056 31923 ts_tablet_manager.cc:1428] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.068s user 0.007s sys 0.000s
I20250831 07:58:14.688326 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:14.708585 31924 ts_tablet_manager.cc:1428] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.083s user 0.008s sys 0.001s
I20250831 07:58:14.709550 31924 tablet_bootstrap.cc:492] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:14.727555 32108 consensus_queue.cc:1035] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 07:58:14.733139 31698 leader_election.cc:343] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.725339 32064 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.743991 32064 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.745486 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:14.745945 31806 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.701886 31923 tablet_bootstrap.cc:492] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:14.749455 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:14.751919 31806 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:14.753222 31699 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:14.754874 32064 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:14.739013 31925 tablet_bootstrap.cc:654] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.755614 32064 raft_consensus.cc:695] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:14.757890 32064 consensus_queue.cc:237] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:14.759578 31698 leader_election.cc:343] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.760639 31924 tablet_bootstrap.cc:654] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
W20250831 07:58:14.765106 31680 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 07:58:14.766269 31925 tablet_bootstrap.cc:492] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:14.767449 31925 ts_tablet_manager.cc:1397] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.086s user 0.007s sys 0.004s
I20250831 07:58:14.767323 32068 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:14.771489 31923 tablet_bootstrap.cc:654] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.770052 32068 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.773726 31925 raft_consensus.cc:357] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.774766 31925 raft_consensus.cc:383] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.775225 31925 raft_consensus.cc:738] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.775981 31925 consensus_queue.cc:260] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.780052 32068 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.781515 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:14.781761 31925 ts_tablet_manager.cc:1428] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20250831 07:58:14.782759 31925 tablet_bootstrap.cc:492] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:14.784189 31883 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:14.784322 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:14.784909 31733 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:14.785400 31773 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:14.787986 32068 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:14.788341 32068 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:14.788683 32068 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.797062 32146 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:14.797537 32146 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.799365 32146 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.800813 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:14.801651 31807 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:14.802088 31775 leader_election.cc:343] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.804821 31924 tablet_bootstrap.cc:492] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:14.803834 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:14.805375 31924 ts_tablet_manager.cc:1397] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.096s user 0.009s sys 0.003s
W20250831 07:58:14.806098 31773 leader_election.cc:343] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.806557 31773 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:14.795418 31628 catalog_manager.cc:5582] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:14.809060 32146 raft_consensus.cc:2747] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:14.809336 31807 raft_consensus.cc:1273] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:14.810706 31997 consensus_queue.cc:1035] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:14.814221 31925 tablet_bootstrap.cc:654] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.813953 31924 raft_consensus.cc:357] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.814690 31924 raft_consensus.cc:383] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.814980 31924 raft_consensus.cc:738] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.816191 32068 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.818861 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:14.819391 31733 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.822360 31923 tablet_bootstrap.cc:492] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:14.822814 31923 ts_tablet_manager.cc:1397] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.121s user 0.009s sys 0.003s
I20250831 07:58:14.824047 32064 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:14.824479 32064 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.825224 31923 raft_consensus.cc:357] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.825932 31923 raft_consensus.cc:383] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.826191 31923 raft_consensus.cc:738] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.826251 31733 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:14.826843 31923 consensus_queue.cc:260] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.827350 31775 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:14.827883 32064 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.828222 32146 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:14.828676 32146 raft_consensus.cc:695] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:14.815642 31924 consensus_queue.cc:260] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.829319 31732 raft_consensus.cc:1273] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:14.829447 32146 consensus_queue.cc:237] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.830646 31924 ts_tablet_manager.cc:1428] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20250831 07:58:14.831545 31924 tablet_bootstrap.cc:492] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:14.831379 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:14.832043 31880 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:14.832592 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:14.833173 31807 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:14.833189 31698 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:14.836035 31923 ts_tablet_manager.cc:1428] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20250831 07:58:14.836174 31997 consensus_queue.cc:1035] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:14.836925 31923 tablet_bootstrap.cc:492] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:14.834126 32064 raft_consensus.cc:2802] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:14.838843 32064 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:14.839200 32064 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.849584 32064 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.857903 32064 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.858438 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:14.859110 31807 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.860169 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:14.860759 31880 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.862152 31923 tablet_bootstrap.cc:654] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.870049 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:14.870697 31883 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.872320 31880 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:14.862172 32068 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.915100 32146 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:14.915633 32146 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.916769 31807 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:14.918036 31699 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:14.918488 31924 tablet_bootstrap.cc:654] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.918857 32064 raft_consensus.cc:2802] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:14.918377 31628 catalog_manager.cc:5582] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:14.919373 32064 raft_consensus.cc:695] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:14.920322 32064 consensus_queue.cc:237] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.925627 32146 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.928093 31923 tablet_bootstrap.cc:492] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:14.928586 31923 ts_tablet_manager.cc:1397] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.092s user 0.017s sys 0.000s
I20250831 07:58:14.931378 31923 raft_consensus.cc:357] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.932094 31923 raft_consensus.cc:383] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.932323 32146 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:14.932411 31923 raft_consensus.cc:738] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.933442 31923 consensus_queue.cc:260] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.933033 32146 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.935101 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:14.936623 32146 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.938277 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:14.938607 31923 ts_tablet_manager.cc:1428] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250831 07:58:14.945225 31923 tablet_bootstrap.cc:492] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:14.949980 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:14.950604 31880 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:14.952311 31924 tablet_bootstrap.cc:492] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:14.952762 31924 ts_tablet_manager.cc:1397] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.121s user 0.016s sys 0.018s
I20250831 07:58:14.954282 31883 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:14.957572 31924 raft_consensus.cc:357] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.958411 31924 raft_consensus.cc:383] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.958694 31924 raft_consensus.cc:738] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.959290 31924 consensus_queue.cc:260] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.960837 31923 tablet_bootstrap.cc:654] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
W20250831 07:58:14.964145 31775 leader_election.cc:343] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:14.964805 31773 leader_election.cc:343] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.961519 31924 ts_tablet_manager.cc:1428] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250831 07:58:14.962590 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:14.963330 31925 tablet_bootstrap.cc:492] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:14.965267 31773 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:14.965711 31925 ts_tablet_manager.cc:1397] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.183s user 0.007s sys 0.021s
I20250831 07:58:14.966024 32065 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 07:58:14.966158 31775 leader_election.cc:343] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.967231 31773 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:14.968053 32065 raft_consensus.cc:2802] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:14.968418 32065 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:14.968343 31925 raft_consensus.cc:357] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.968873 32065 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.969084 31925 raft_consensus.cc:383] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:14.969404 31925 raft_consensus.cc:738] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:14.969888 31925 consensus_queue.cc:260] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.971444 31924 tablet_bootstrap.cc:492] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:14.975805 32065 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:14.978818 32065 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:14.979444 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:58:14.981352 31775 leader_election.cc:343] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:14.984375 31925 ts_tablet_manager.cc:1428] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.018s user 0.003s sys 0.002s
I20250831 07:58:14.983966 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:14.984524 31924 tablet_bootstrap.cc:654] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.985554 31883 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:14.986225 31925 tablet_bootstrap.cc:492] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:14.983949 31630 catalog_manager.cc:5582] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:14.996932 31923 tablet_bootstrap.cc:492] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:14.997087 31925 tablet_bootstrap.cc:654] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:14.997395 31923 ts_tablet_manager.cc:1397] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.000s
I20250831 07:58:15.002622 31883 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:15.001883 31732 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:15.007759 31773 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:15.008152 31923 raft_consensus.cc:357] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.008831 31923 raft_consensus.cc:383] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.008919 32067 raft_consensus.cc:2802] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:15.009166 31923 raft_consensus.cc:738] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.009467 32067 raft_consensus.cc:695] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:15.009872 31923 consensus_queue.cc:260] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.010283 32067 consensus_queue.cc:237] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.015740 31923 ts_tablet_manager.cc:1428] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250831 07:58:15.016602 31923 tablet_bootstrap.cc:492] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:15.019284 31924 tablet_bootstrap.cc:492] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:15.019721 31924 ts_tablet_manager.cc:1397] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.007s
I20250831 07:58:15.036790 31924 raft_consensus.cc:357] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.037317 31924 raft_consensus.cc:383] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.037530 31924 raft_consensus.cc:738] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.039026 31924 consensus_queue.cc:260] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.038763 32078 consensus_queue.cc:1035] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250831 07:58:15.042003 31924 ts_tablet_manager.cc:1428] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.022s user 0.004s sys 0.001s
I20250831 07:58:15.042837 31924 tablet_bootstrap.cc:492] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:15.046241 31925 tablet_bootstrap.cc:492] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:15.046703 31925 ts_tablet_manager.cc:1397] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.061s user 0.020s sys 0.010s
I20250831 07:58:15.049090 31925 raft_consensus.cc:357] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.049716 31925 raft_consensus.cc:383] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.050005 31925 raft_consensus.cc:738] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.052220 31628 catalog_manager.cc:5582] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:15.053848 31923 tablet_bootstrap.cc:654] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:15.052685 31925 consensus_queue.cc:260] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.056527 31924 tablet_bootstrap.cc:654] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:15.059414 31925 ts_tablet_manager.cc:1428] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20250831 07:58:15.060353 31925 tablet_bootstrap.cc:492] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:15.073375 31925 tablet_bootstrap.cc:654] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:15.075321 32078 consensus_queue.cc:1035] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:15.080524 31924 tablet_bootstrap.cc:492] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:15.081003 31924 ts_tablet_manager.cc:1397] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.006s
I20250831 07:58:15.085841 31924 raft_consensus.cc:357] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.086494 31924 raft_consensus.cc:383] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.086715 31924 raft_consensus.cc:738] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.087186 31924 consensus_queue.cc:260] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.088656 32078 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:15.089800 32078 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.099602 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:15.100344 31732 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:15.101560 31849 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:15.103662 31997 raft_consensus.cc:2802] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:15.103979 32078 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:15.104157 31924 ts_tablet_manager.cc:1428] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20250831 07:58:15.105271 31924 tablet_bootstrap.cc:492] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:15.106246 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:15.106787 31925 tablet_bootstrap.cc:492] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:15.106870 31807 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:15.107313 31925 ts_tablet_manager.cc:1397] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.006s
I20250831 07:58:15.104043 31997 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:15.108829 31997 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:15.111577 31925 raft_consensus.cc:357] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.112247 31925 raft_consensus.cc:383] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.112478 31925 raft_consensus.cc:738] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.113003 31923 tablet_bootstrap.cc:492] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:15.113420 31923 ts_tablet_manager.cc:1397] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.097s user 0.012s sys 0.006s
I20250831 07:58:15.113143 31925 consensus_queue.cc:260] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.114094 31924 tablet_bootstrap.cc:654] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:15.116623 31923 raft_consensus.cc:357] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.117309 31923 raft_consensus.cc:383] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.117560 31923 raft_consensus.cc:738] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.118196 31923 consensus_queue.cc:260] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.121107 31923 ts_tablet_manager.cc:1428] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250831 07:58:15.123798 31923 tablet_bootstrap.cc:492] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:15.133610 32013 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:15.134198 32013 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.138006 32013 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:15.138954 31997 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.139663 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:15.140820 31997 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:15.142551 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:15.143216 31883 raft_consensus.cc:2391] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:15.143293 32146 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:15.143791 32146 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.145246 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:15.145835 32146 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:15.145884 31808 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:15.149641 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:15.151189 31698 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:15.149641 31923 tablet_bootstrap.cc:654] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:15.152094 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:15.152657 31733 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:15.154065 32151 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:15.157608 31775 leader_election.cc:343] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:15.162006 31925 ts_tablet_manager.cc:1428] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.054s user 0.004s sys 0.004s
I20250831 07:58:15.162612 32014 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:15.163071 32014 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.163962 31923 tablet_bootstrap.cc:492] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:15.164424 31923 ts_tablet_manager.cc:1397] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.002s
I20250831 07:58:15.164945 32014 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:15.167136 31923 raft_consensus.cc:357] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.167856 31923 raft_consensus.cc:383] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.168190 31923 raft_consensus.cc:738] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.169050 31923 consensus_queue.cc:260] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.167583 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:15.171370 31923 ts_tablet_manager.cc:1428] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 07:58:15.173064 31923 tablet_bootstrap.cc:492] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:15.173182 31924 tablet_bootstrap.cc:492] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:15.173659 31924 ts_tablet_manager.cc:1397] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.011s
I20250831 07:58:15.176476 31925 tablet_bootstrap.cc:492] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
W20250831 07:58:15.176550 31773 leader_election.cc:343] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:15.177139 31773 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:15.178404 32146 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:15.178921 31924 raft_consensus.cc:357] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.179586 31924 raft_consensus.cc:383] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.179875 31924 raft_consensus.cc:738] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.180464 31924 consensus_queue.cc:260] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.183039 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:15.192299 31924 ts_tablet_manager.cc:1428] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20250831 07:58:15.193091 31924 tablet_bootstrap.cc:492] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:15.193825 31808 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:15.194940 31997 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:15.197536 31997 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.196527 31733 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:15.200975 32151 raft_consensus.cc:2747] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 07:58:15.195253 31698 leader_election.cc:343] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:15.200623 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:15.198112 31925 tablet_bootstrap.cc:654] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:15.201706 31849 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:15.201973 31997 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:15.203629 31849 leader_election.cc:343] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:15.203759 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:15.204888 31997 raft_consensus.cc:2802] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
W20250831 07:58:15.205027 31699 leader_election.cc:343] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:15.205510 31699 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:15.208019 31924 tablet_bootstrap.cc:654] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:15.209409 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:15.215934 31923 tablet_bootstrap.cc:654] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:15.206298 32151 raft_consensus.cc:2747] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:15.219506 31997 raft_consensus.cc:695] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:15.220525 31997 consensus_queue.cc:237] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.228557 31925 tablet_bootstrap.cc:492] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:15.229089 31925 ts_tablet_manager.cc:1397] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.053s user 0.016s sys 0.007s
W20250831 07:58:15.234301 31848 leader_election.cc:343] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:15.234774 31848 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:15.235687 31997 raft_consensus.cc:2747] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:15.237459 31925 raft_consensus.cc:357] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.238168 31925 raft_consensus.cc:383] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.238431 31925 raft_consensus.cc:738] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.239241 31925 consensus_queue.cc:260] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.249488 32065 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:15.250288 31630 catalog_manager.cc:5582] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:15.250190 32065 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.260828 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:15.261683 31732 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:15.263152 31775 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:15.264307 32067 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:15.264712 32067 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:15.265060 32067 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:15.265254 31924 tablet_bootstrap.cc:492] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:15.265506 31997 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:15.265717 31924 ts_tablet_manager.cc:1397] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.073s user 0.011s sys 0.009s
I20250831 07:58:15.266021 31997 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.268222 31924 raft_consensus.cc:357] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.268875 31924 raft_consensus.cc:383] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.268756 32065 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:15.269250 31924 raft_consensus.cc:738] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.269869 31924 consensus_queue.cc:260] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.268636 31997 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:15.270982 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:15.271612 31732 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:15.272653 31849 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:15.273238 31923 tablet_bootstrap.cc:492] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:15.273669 31923 ts_tablet_manager.cc:1397] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.101s user 0.010s sys 0.013s
I20250831 07:58:15.275013 31997 raft_consensus.cc:2802] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:15.275378 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:15.275784 31997 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:15.276324 31997 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:15.277078 31924 ts_tablet_manager.cc:1428] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250831 07:58:15.277941 31924 tablet_bootstrap.cc:492] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:15.283396 31924 tablet_bootstrap.cc:654] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:15.276142 31923 raft_consensus.cc:357] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.283962 31923 raft_consensus.cc:383] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.284235 31923 raft_consensus.cc:738] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.276602 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:15.284791 31807 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:15.284803 31923 consensus_queue.cc:260] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.286623 31923 ts_tablet_manager.cc:1428] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250831 07:58:15.287482 31923 tablet_bootstrap.cc:492] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:15.295637 31923 tablet_bootstrap.cc:654] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
W20250831 07:58:15.296200 31773 leader_election.cc:343] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:15.307552 31925 ts_tablet_manager.cc:1428] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.078s user 0.012s sys 0.012s
I20250831 07:58:15.308540 31925 tablet_bootstrap.cc:492] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:15.311152 32065 consensus_queue.cc:1035] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:15.313791 32067 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.319752 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:15.320520 31732 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:15.315726 32067 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:15.326941 32064 consensus_queue.cc:1035] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:15.337515 31997 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.350348 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:15.340673 32064 consensus_queue.cc:1035] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250831 07:58:15.357842 32065 consensus_queue.cc:1035] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:15.362349 31881 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:15.376701 31773 leader_election.cc:343] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:15.378171 31732 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:15.382490 31775 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:15.385128 32065 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:15.388899 32065 raft_consensus.cc:695] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:15.389835 32078 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:15.389725 32065 consensus_queue.cc:237] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.390288 32078 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.391192 32014 consensus_queue.cc:1035] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:15.394658 32078 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:15.406154 31997 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:15.409027 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:15.395802 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:15.416162 31925 tablet_bootstrap.cc:654] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:15.420603 31924 tablet_bootstrap.cc:492] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:15.421334 31924 ts_tablet_manager.cc:1397] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.144s user 0.021s sys 0.006s
I20250831 07:58:15.422240 31923 tablet_bootstrap.cc:492] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:15.427959 31923 ts_tablet_manager.cc:1397] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.141s user 0.016s sys 0.010s
I20250831 07:58:15.416713 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:15.416183 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:15.441285 31849 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:15.440536 31728 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:15.598925 31914 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.IsCreateTableDone -> {remote=127.29.217.62:40065, user_credentials={real_user=slave}} blocked reactor thread for 109854us
I20250831 07:58:15.608963 31728 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:15.440810 31807 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:15.599750 31923 raft_consensus.cc:357] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.610385 31923 raft_consensus.cc:383] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.610673 31923 raft_consensus.cc:738] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.611275 31923 consensus_queue.cc:260] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.438212 32065 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:15.616546 31924 raft_consensus.cc:357] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.617377 31924 raft_consensus.cc:383] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.617650 31924 raft_consensus.cc:738] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.618218 31924 consensus_queue.cc:260] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.638178 32108 raft_consensus.cc:2747] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:15.655480 32068 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:15.660012 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 218792us
I20250831 07:58:15.630616 32065 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.666085 32151 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:15.692845 31923 ts_tablet_manager.cc:1428] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.264s user 0.020s sys 0.015s
I20250831 07:58:15.679230 32068 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.702359 32068 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:15.700291 32214 log.cc:927] Time spent T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250831 07:58:15.704409 31924 ts_tablet_manager.cc:1428] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.283s user 0.023s sys 0.030s
I20250831 07:58:15.705680 31924 tablet_bootstrap.cc:492] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
W20250831 07:58:15.711073 32212 log.cc:927] Time spent T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20250831 07:58:15.712404 31807 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:15.715039 31881 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:15.702742 32065 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:15.726572 31848 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:15.725543 31631 catalog_manager.cc:5582] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
W20250831 07:58:15.796486 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 69999.3us
I20250831 07:58:15.688071 32151 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:15.808406 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 82256.3us
I20250831 07:58:15.801952 32222 consensus_queue.cc:1035] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.086s
I20250831 07:58:15.801923 31954 raft_consensus.cc:2802] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:15.842744 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:15.888450 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:15.892068 32151 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:15.892189 31773 leader_election.cc:343] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:15.897541 31954 raft_consensus.cc:695] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:15.908878 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:15.914597 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:15.922253 31925 tablet_bootstrap.cc:492] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:15.921943 31954 consensus_queue.cc:237] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.922699 31925 ts_tablet_manager.cc:1397] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.614s user 0.007s sys 0.016s
I20250831 07:58:15.930311 31925 raft_consensus.cc:357] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.930951 31925 raft_consensus.cc:383] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:15.931246 31925 raft_consensus.cc:738] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:15.931802 31925 consensus_queue.cc:260] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:15.937059 31773 leader_election.cc:343] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:15.948302 31925 ts_tablet_manager.cc:1428] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20250831 07:58:15.949056 31925 tablet_bootstrap.cc:492] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:15.953799 32220 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:15.954286 32220 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:15.959291 31923 tablet_bootstrap.cc:492] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
W20250831 07:58:15.979970 32213 log.cc:927] Time spent T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.377s user 0.004s sys 0.000s
I20250831 07:58:15.983920 32220 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:15.988413 31923 tablet_bootstrap.cc:654] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:15.995736 31924 tablet_bootstrap.cc:654] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:15.998653 32067 consensus_queue.cc:1035] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 07:58:16.027355 31775 leader_election.cc:343] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.028205 31775 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:16.028576 31773 consensus_peers.cc:489] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 -> Peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Couldn't send request to peer 8247a38b550649aa9cd6e7668dcaec10. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250831 07:58:16.029434 31775 leader_election.cc:343] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.029815 31775 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:16.031709 32218 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:16.035082 32225 consensus_queue.cc:1035] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250831 07:58:16.036229 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:16.037005 31808 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:16.046129 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:16.157377 31880 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
W20250831 07:58:16.033974 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 82696.6us
I20250831 07:58:16.033322 32067 raft_consensus.cc:2747] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:16.170037 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:16.136545 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:16.181990 31699 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:16.182890 32220 raft_consensus.cc:2802] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:16.183247 32220 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:16.183566 32220 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:16.195497 31923 tablet_bootstrap.cc:492] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:16.196071 31923 ts_tablet_manager.cc:1397] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.237s user 0.010s sys 0.007s
I20250831 07:58:16.183993 31954 consensus_queue.cc:1035] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 07:58:16.204545 31699 leader_election.cc:343] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.210501 31923 raft_consensus.cc:357] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.211483 31923 raft_consensus.cc:383] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:16.211870 31923 raft_consensus.cc:738] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:16.212761 31923 consensus_queue.cc:260] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.219213 31925 tablet_bootstrap.cc:654] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
W20250831 07:58:16.223088 31698 leader_election.cc:343] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.223543 31698 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:16.228856 32014 raft_consensus.cc:2747] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:16.230422 31954 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:16.239400 31923 ts_tablet_manager.cc:1428] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.043s user 0.006s sys 0.001s
I20250831 07:58:16.241133 31923 tablet_bootstrap.cc:492] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:16.239841 31954 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.252687 31954 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:16.293094 31680 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 07:58:16.294190 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:16.295120 31924 tablet_bootstrap.cc:492] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:16.299722 31997 consensus_queue.cc:1035] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:16.301578 31924 ts_tablet_manager.cc:1397] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.596s user 0.028s sys 0.014s
I20250831 07:58:16.313314 32229 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:16.313603 32068 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:16.313851 32229 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.315801 32229 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:16.321192 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:16.321798 31734 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:16.325280 32220 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.340392 32220 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:16.340561 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:16.342285 31803 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:16.345268 31848 leader_election.cc:343] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.345301 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:16.347502 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:16.348073 31882 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:16.353190 31849 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:16.354463 32236 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:16.354841 32236 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:16.355155 32236 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:16.358893 31882 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:16.359921 31698 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:16.362486 32230 raft_consensus.cc:2802] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:16.363173 32230 raft_consensus.cc:695] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:58:16.381764 31849 leader_election.cc:343] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.353220 31803 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:16.380611 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:16.420428 31809 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:16.422504 31848 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:16.428404 32229 raft_consensus.cc:2802] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:16.445721 31925 tablet_bootstrap.cc:492] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:16.428160 32230 consensus_queue.cc:237] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.444556 32229 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:16.448509 31925 ts_tablet_manager.cc:1397] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.500s user 0.006s sys 0.024s
I20250831 07:58:16.453261 32229 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:16.456308 32236 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.459729 32236 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:16.465696 32229 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.466082 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:16.466899 31733 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:16.467634 31923 tablet_bootstrap.cc:654] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:16.468636 31924 raft_consensus.cc:357] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.475046 31924 raft_consensus.cc:383] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:16.475492 31924 raft_consensus.cc:738] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:16.478401 31924 consensus_queue.cc:260] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.480319 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:58:16.490798 31848 leader_election.cc:343] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.493042 32229 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:16.493849 31925 raft_consensus.cc:357] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.504567 31925 raft_consensus.cc:383] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:16.504866 31925 raft_consensus.cc:738] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:16.506349 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:16.496703 31924 ts_tablet_manager.cc:1428] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.195s user 0.012s sys 0.008s
I20250831 07:58:16.498173 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:16.508370 31805 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:16.508567 31849 leader_election.cc:343] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.493876 32229 consensus_queue.cc:1035] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20250831 07:58:16.510330 31924 tablet_bootstrap.cc:492] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:16.507340 31925 consensus_queue.cc:260] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.501250 31733 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:16.518244 31849 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:16.523149 31925 ts_tablet_manager.cc:1428] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.073s user 0.000s sys 0.008s
I20250831 07:58:16.521265 31631 catalog_manager.cc:5582] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:16.526643 31925 tablet_bootstrap.cc:492] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:16.533856 31805 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:16.534813 31848 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:16.525314 32229 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:16.536258 31954 raft_consensus.cc:2802] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:16.536995 32221 consensus_queue.cc:1035] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:16.546005 31628 catalog_manager.cc:5582] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:16.548970 31954 raft_consensus.cc:695] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:16.569729 31924 tablet_bootstrap.cc:654] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:16.570976 31923 tablet_bootstrap.cc:492] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:16.573380 31923 ts_tablet_manager.cc:1397] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.332s user 0.016s sys 0.025s
I20250831 07:58:16.562736 31954 consensus_queue.cc:237] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.586838 31925 tablet_bootstrap.cc:654] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:16.581542 31923 raft_consensus.cc:357] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.591879 31923 raft_consensus.cc:383] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:16.592765 31923 raft_consensus.cc:738] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:16.548246 32229 raft_consensus.cc:695] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:16.593698 31923 consensus_queue.cc:260] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.595324 31923 ts_tablet_manager.cc:1428] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20250831 07:58:16.594424 32229 consensus_queue.cc:237] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.596276 31923 tablet_bootstrap.cc:492] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:16.607313 31924 tablet_bootstrap.cc:492] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:16.607780 31924 ts_tablet_manager.cc:1397] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.098s user 0.009s sys 0.003s
I20250831 07:58:16.610219 31924 raft_consensus.cc:357] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.610827 31924 raft_consensus.cc:383] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:16.611071 31924 raft_consensus.cc:738] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:16.611641 31924 consensus_queue.cc:260] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.622151 31924 ts_tablet_manager.cc:1428] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250831 07:58:16.623067 31924 tablet_bootstrap.cc:492] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:16.628654 31923 tablet_bootstrap.cc:654] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:16.639286 31925 tablet_bootstrap.cc:492] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:16.639786 31925 ts_tablet_manager.cc:1397] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.115s user 0.005s sys 0.010s
I20250831 07:58:16.642328 31925 raft_consensus.cc:357] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.642989 31925 raft_consensus.cc:383] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:16.643239 31925 raft_consensus.cc:738] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:16.644219 31924 tablet_bootstrap.cc:654] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:16.647230 31925 consensus_queue.cc:260] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.653092 31925 ts_tablet_manager.cc:1428] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250831 07:58:16.661078 31925 tablet_bootstrap.cc:492] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:16.664000 31923 tablet_bootstrap.cc:492] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:16.664530 31923 ts_tablet_manager.cc:1397] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.068s user 0.015s sys 0.006s
I20250831 07:58:16.667059 31925 tablet_bootstrap.cc:654] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:16.666870 31923 raft_consensus.cc:357] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.667534 31923 raft_consensus.cc:383] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:16.667811 31923 raft_consensus.cc:738] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:16.669988 31924 tablet_bootstrap.cc:492] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:16.670488 31924 ts_tablet_manager.cc:1397] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.003s
I20250831 07:58:16.673243 31924 raft_consensus.cc:357] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.673915 31924 raft_consensus.cc:383] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:16.674206 31924 raft_consensus.cc:738] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:16.674799 31924 consensus_queue.cc:260] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.682523 31924 ts_tablet_manager.cc:1428] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20250831 07:58:16.684252 31924 tablet_bootstrap.cc:492] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:16.688712 31923 consensus_queue.cc:260] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.689388 31631 catalog_manager.cc:5582] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:16.690460 31924 tablet_bootstrap.cc:654] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:16.691504 31923 ts_tablet_manager.cc:1428] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250831 07:58:16.690927 31631 catalog_manager.cc:5582] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:16.691632 31925 tablet_bootstrap.cc:492] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:16.692349 31925 ts_tablet_manager.cc:1397] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.007s
I20250831 07:58:16.692373 31923 tablet_bootstrap.cc:492] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:16.699198 31923 tablet_bootstrap.cc:654] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:16.704761 32068 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:16.703249 31925 raft_consensus.cc:357] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.705433 31925 raft_consensus.cc:383] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:16.705226 32068 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.706027 31925 raft_consensus.cc:738] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:16.707185 32068 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:16.706934 31925 consensus_queue.cc:260] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.708622 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:16.708701 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:16.709261 31733 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
W20250831 07:58:16.710896 31773 leader_election.cc:343] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.711964 31775 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:16.713017 32217 raft_consensus.cc:2802] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:16.713256 31925 ts_tablet_manager.cc:1428] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20250831 07:58:16.713399 32217 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:16.715238 32217 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:16.725462 31923 tablet_bootstrap.cc:492] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:16.726016 31923 ts_tablet_manager.cc:1397] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.011s
I20250831 07:58:16.728526 32217 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.730580 31925 tablet_bootstrap.cc:492] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:16.731231 31923 raft_consensus.cc:357] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.734264 31923 raft_consensus.cc:383] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:16.734532 31923 raft_consensus.cc:738] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:16.735186 31923 consensus_queue.cc:260] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.737649 31923 ts_tablet_manager.cc:1428] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250831 07:58:16.738586 31923 tablet_bootstrap.cc:492] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:16.738602 32223 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:16.739277 31925 tablet_bootstrap.cc:654] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:16.739238 32223 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.741488 32223 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:16.743863 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:16.744179 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:16.745388 31923 tablet_bootstrap.cc:654] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
W20250831 07:58:16.745302 31698 leader_election.cc:343] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:16.746309 31699 leader_election.cc:343] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.746737 31699 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:16.748272 32230 raft_consensus.cc:2747] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:16.748251 32223 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:16.749425 32223 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.753141 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:16.754014 31803 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:16.755545 31699 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:16.760921 32230 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:16.761519 32230 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:16.761883 32230 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:16.778376 32217 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:16.780262 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:16.781409 31733 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:16.788690 31924 tablet_bootstrap.cc:492] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:16.789301 31924 ts_tablet_manager.cc:1397] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.105s user 0.018s sys 0.019s
I20250831 07:58:16.794035 31954 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:16.794507 31954 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.794935 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:58:16.805619 31773 leader_election.cc:343] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.807096 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:16.806896 32230 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.807897 31954 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:16.808375 31698 leader_election.cc:343] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.808800 32230 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:16.808909 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:16.810866 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:16.811743 31733 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:16.812943 31775 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:16.803514 32223 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:16.811656 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:16.814424 32217 raft_consensus.cc:2802] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
W20250831 07:58:16.814327 31698 leader_election.cc:343] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.818475 31923 tablet_bootstrap.cc:492] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:16.819128 31923 ts_tablet_manager.cc:1397] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.081s user 0.022s sys 0.023s
I20250831 07:58:16.818703 31924 raft_consensus.cc:357] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.820749 31924 raft_consensus.cc:383] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:16.821030 31924 raft_consensus.cc:738] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:16.815734 31803 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:16.821969 31924 consensus_queue.cc:260] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.816560 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:16.824366 31925 tablet_bootstrap.cc:492] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:16.824803 31925 ts_tablet_manager.cc:1397] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.094s user 0.025s sys 0.029s
I20250831 07:58:16.820276 32217 raft_consensus.cc:695] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:16.831990 31848 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:16.832355 32217 consensus_queue.cc:237] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.832445 31925 raft_consensus.cc:357] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.833227 31925 raft_consensus.cc:383] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:16.833544 31925 raft_consensus.cc:738] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:16.834131 31925 consensus_queue.cc:260] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.836512 32223 consensus_queue.cc:1035] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:16.839962 32229 raft_consensus.cc:2747] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:16.841777 31803 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:16.843431 31699 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:16.844777 31925 ts_tablet_manager.cc:1428] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20250831 07:58:16.844977 32067 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:16.844993 31923 raft_consensus.cc:357] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.845733 31925 tablet_bootstrap.cc:492] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:16.845818 31923 raft_consensus.cc:383] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:16.847150 31923 raft_consensus.cc:738] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:16.845553 32067 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.847810 31923 consensus_queue.cc:260] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.848960 31924 ts_tablet_manager.cc:1428] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.059s user 0.006s sys 0.000s
I20250831 07:58:16.849599 32067 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:16.851436 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:16.854512 31924 tablet_bootstrap.cc:492] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:16.852185 31878 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:16.855777 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:16.857208 31773 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:16.858440 31923 ts_tablet_manager.cc:1428] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.039s user 0.008s sys 0.000s
I20250831 07:58:16.859328 31923 tablet_bootstrap.cc:492] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:16.857999 32068 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250831 07:58:16.860236 31775 leader_election.cc:343] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.862401 32229 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:16.862463 32068 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:16.862965 32229 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.862951 32068 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:16.865046 32229 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:16.850589 32223 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:16.865963 32223 raft_consensus.cc:695] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:16.866873 32223 consensus_queue.cc:237] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.868505 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:16.869876 31849 leader_election.cc:343] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.874408 31925 tablet_bootstrap.cc:654] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:16.874939 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:16.874953 32068 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:16.876384 31848 leader_election.cc:343] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.876750 31848 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:16.877629 32068 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:16.878132 31954 raft_consensus.cc:2747] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:16.878295 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:16.878855 31877 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:16.881078 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:58:16.884584 31775 leader_election.cc:343] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.893894 31924 tablet_bootstrap.cc:654] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:16.901494 31629 catalog_manager.cc:5582] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:16.902812 31877 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:16.904799 31773 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:16.910180 31878 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:16.907769 32217 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:16.912642 32217 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:16.921056 31923 tablet_bootstrap.cc:654] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:16.922694 32217 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:16.927151 32231 consensus_queue.cc:1035] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:16.930797 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:16.933066 31773 leader_election.cc:343] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.933969 32068 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:16.940150 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:16.968838 32068 raft_consensus.cc:695] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:16.969839 32068 consensus_queue.cc:237] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:16.975590 31775 leader_election.cc:343] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:16.976177 31775 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:16.986774 32067 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:16.995126 31924 tablet_bootstrap.cc:492] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:16.997246 31924 ts_tablet_manager.cc:1397] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.143s user 0.005s sys 0.008s
I20250831 07:58:16.996547 32217 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:17.000479 31923 tablet_bootstrap.cc:492] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:17.001006 31923 ts_tablet_manager.cc:1397] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.142s user 0.023s sys 0.011s
I20250831 07:58:17.000535 32217 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.007948 31924 raft_consensus.cc:357] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.008875 31924 raft_consensus.cc:383] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.009209 31924 raft_consensus.cc:738] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.010012 31924 consensus_queue.cc:260] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.062055 31924 ts_tablet_manager.cc:1428] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.064s user 0.009s sys 0.000s
I20250831 07:58:17.063055 31925 tablet_bootstrap.cc:492] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:17.064225 31925 ts_tablet_manager.cc:1397] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.219s user 0.034s sys 0.071s
I20250831 07:58:17.051092 31630 catalog_manager.cc:5582] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:17.077857 32217 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:17.081248 31924 tablet_bootstrap.cc:492] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:17.081166 31923 raft_consensus.cc:357] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.082340 31923 raft_consensus.cc:383] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.082729 31923 raft_consensus.cc:738] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.079107 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:17.084039 31734 raft_consensus.cc:2391] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 1: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:17.083508 31923 consensus_queue.cc:260] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.091420 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:17.093693 32231 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:17.098466 31925 raft_consensus.cc:357] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.103327 31925 raft_consensus.cc:383] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.103927 31925 raft_consensus.cc:738] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.106370 31923 ts_tablet_manager.cc:1428] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.105s user 0.010s sys 0.004s
I20250831 07:58:17.109668 31925 consensus_queue.cc:260] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.094564 32231 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.113206 32229 consensus_queue.cc:1035] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:17.115161 32231 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:17.119555 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:17.120910 31877 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:17.121968 31923 tablet_bootstrap.cc:492] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:17.141166 32216 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:17.142692 31924 tablet_bootstrap.cc:654] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.141999 32216 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.143327 31698 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:17.144265 32230 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:17.144634 32230 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:17.144714 32216 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:17.145017 32230 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.148658 31925 ts_tablet_manager.cc:1428] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.084s user 0.010s sys 0.002s
I20250831 07:58:17.152724 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:17.154289 31923 tablet_bootstrap.cc:654] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.160166 31806 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.161967 31878 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:17.161844 32229 consensus_queue.cc:1035] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:17.166698 31925 tablet_bootstrap.cc:492] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:17.168910 31773 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:17.169950 32217 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:17.172080 32217 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:17.172492 32217 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.181843 32221 consensus_queue.cc:1035] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:17.192281 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:17.192441 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:17.197458 31630 catalog_manager.cc:5582] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:17.206903 32217 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.207952 31806 raft_consensus.cc:2391] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:17.182052 32230 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.225987 32230 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:17.228145 31733 raft_consensus.cc:2391] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 1: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:17.227805 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:17.228734 31877 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.237529 31775 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:17.239624 32067 raft_consensus.cc:2747] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 07:58:17.241773 31849 consensus_peers.cc:489] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 -> Peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Couldn't send request to peer a0d845d01c8c4894895f66e3beae298c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:17.245838 32217 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:17.250784 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:17.251796 31877 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:17.254272 32223 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:17.254839 32223 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.248368 32231 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:17.261660 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:17.261866 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:17.262477 31733 raft_consensus.cc:2391] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 1: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:17.262735 31802 raft_consensus.cc:2391] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:17.259130 32223 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:17.266618 32231 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.268179 31698 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:17.268661 32231 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:17.268762 32230 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:17.269331 32230 raft_consensus.cc:695] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:17.271646 32230 consensus_queue.cc:237] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.272698 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:17.278148 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:17.283337 31698 leader_election.cc:343] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.284188 31775 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:17.285054 32068 raft_consensus.cc:2747] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250831 07:58:17.286648 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:17.291291 31698 leader_election.cc:343] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:17.293285 31773 consensus_peers.cc:489] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 -> Peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Couldn't send request to peer 8247a38b550649aa9cd6e7668dcaec10. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:17.295542 31925 tablet_bootstrap.cc:654] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.308233 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:17.328200 32228 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:17.328848 31699 leader_election.cc:343] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.328701 32228 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.329282 31699 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:17.329739 31924 tablet_bootstrap.cc:492] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:17.330207 31924 ts_tablet_manager.cc:1397] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.249s user 0.012s sys 0.007s
I20250831 07:58:17.330909 32228 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:17.332814 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:17.333478 31699 leader_election.cc:343] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.333730 31923 tablet_bootstrap.cc:492] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:17.333906 31699 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:17.334228 31923 ts_tablet_manager.cc:1397] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.213s user 0.006s sys 0.014s
I20250831 07:58:17.333999 31809 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:17.335036 32231 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:17.336679 32231 raft_consensus.cc:2747] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:17.336755 31924 raft_consensus.cc:357] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.337586 31924 raft_consensus.cc:383] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.337953 31924 raft_consensus.cc:738] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.339257 31924 consensus_queue.cc:260] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.341311 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:17.341886 32230 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:17.341990 31732 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:17.343451 31849 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:17.345640 31923 raft_consensus.cc:357] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.346382 31923 raft_consensus.cc:383] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.346666 31923 raft_consensus.cc:738] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.348052 32228 raft_consensus.cc:2802] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:17.343434 31924 ts_tablet_manager.cc:1428] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20250831 07:58:17.348428 32228 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:17.349381 32228 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.342459 32230 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.351390 31924 tablet_bootstrap.cc:492] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:17.347477 31923 consensus_queue.cc:260] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.359274 32228 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.371408 32228 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:17.373165 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:17.373697 31806 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.374230 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:17.374807 31732 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.358794 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:17.359750 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:17.359969 32230 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:17.378901 31877 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
W20250831 07:58:17.380106 31699 leader_election.cc:343] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.380601 31698 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:17.382120 31806 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:17.382454 31732 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:17.382619 32230 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:17.383706 31848 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:17.384142 32230 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:17.384588 32236 raft_consensus.cc:2802] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:17.384878 32230 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.385138 32236 raft_consensus.cc:695] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:17.386335 32236 consensus_queue.cc:237] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.390758 31924 tablet_bootstrap.cc:654] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.391175 31923 ts_tablet_manager.cc:1428] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.057s user 0.007s sys 0.000s
I20250831 07:58:17.392300 31923 tablet_bootstrap.cc:492] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:17.392469 32230 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.396608 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:17.397241 31877 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.395655 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:17.401738 31925 tablet_bootstrap.cc:492] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:17.402196 31925 ts_tablet_manager.cc:1397] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.236s user 0.006s sys 0.007s
W20250831 07:58:17.402379 31699 leader_election.cc:343] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.404695 31925 raft_consensus.cc:357] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.405340 31925 raft_consensus.cc:383] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.405607 31925 raft_consensus.cc:738] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.408898 31923 tablet_bootstrap.cc:654] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.412072 31877 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:17.413175 31698 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:17.413965 32223 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:17.414427 32223 raft_consensus.cc:695] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:17.415218 32223 consensus_queue.cc:237] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.417328 31924 tablet_bootstrap.cc:492] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:17.423489 31924 ts_tablet_manager.cc:1397] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.007s
I20250831 07:58:17.417749 31925 consensus_queue.cc:260] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.418872 32230 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:17.428844 31925 ts_tablet_manager.cc:1428] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.026s user 0.002s sys 0.005s
I20250831 07:58:17.430167 31925 tablet_bootstrap.cc:492] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
W20250831 07:58:17.434738 31698 consensus_peers.cc:489] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c -> Peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Couldn't send request to peer 8247a38b550649aa9cd6e7668dcaec10. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:17.439064 31924 raft_consensus.cc:357] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.439728 31924 raft_consensus.cc:383] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.440029 31924 raft_consensus.cc:738] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.440354 31923 tablet_bootstrap.cc:492] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:17.440861 31923 ts_tablet_manager.cc:1397] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.049s user 0.024s sys 0.008s
I20250831 07:58:17.441085 31924 consensus_queue.cc:260] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.443311 31923 raft_consensus.cc:357] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.445752 31923 raft_consensus.cc:383] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.446039 31923 raft_consensus.cc:738] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
W20250831 07:58:17.446893 31680 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 07:58:17.446645 31923 consensus_queue.cc:260] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.447706 32216 consensus_queue.cc:1035] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:17.449393 31923 ts_tablet_manager.cc:1428] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250831 07:58:17.450374 31923 tablet_bootstrap.cc:492] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:17.454460 31925 tablet_bootstrap.cc:654] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.456209 31923 tablet_bootstrap.cc:654] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.458308 31924 ts_tablet_manager.cc:1428] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.034s user 0.004s sys 0.003s
I20250831 07:58:17.459592 31924 tablet_bootstrap.cc:492] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:17.476850 31630 catalog_manager.cc:5582] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:17.466488 32231 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250831 07:58:17.497634 32236 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:17.498353 32236 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.517172 31924 tablet_bootstrap.cc:654] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.529556 32068 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:17.530226 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:17.530834 31727 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:17.533910 32236 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:17.535246 31923 tablet_bootstrap.cc:492] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:17.536432 31923 ts_tablet_manager.cc:1397] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.086s user 0.009s sys 0.017s
I20250831 07:58:17.543903 31849 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:17.537420 32216 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:17.546500 32221 raft_consensus.cc:2802] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:17.546883 32221 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:17.547219 32221 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.547761 32229 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:17.546373 32216 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.548342 32229 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.550249 32229 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:17.551365 32216 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:17.552315 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:17.553592 31848 leader_election.cc:343] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.554401 31925 tablet_bootstrap.cc:492] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:17.555361 31727 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.544036 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:17.556376 31925 ts_tablet_manager.cc:1397] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.127s user 0.009s sys 0.013s
I20250831 07:58:17.556576 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:17.557087 31733 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
W20250831 07:58:17.557921 31848 leader_election.cc:343] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.559975 31849 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:17.560461 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:17.561741 32236 raft_consensus.cc:2802] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:17.562181 32236 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:17.562522 32236 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.578783 31877 raft_consensus.cc:2391] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 1: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:17.577654 32221 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.583693 32217 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:17.584086 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:17.585461 31775 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:17.586464 32217 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.590186 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:17.592706 32221 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:17.594396 32217 raft_consensus.cc:2747] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:17.603353 31925 raft_consensus.cc:357] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.605048 32236 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.611246 31925 raft_consensus.cc:383] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.612545 31925 raft_consensus.cc:738] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.612262 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:17.613071 31925 consensus_queue.cc:260] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:17.617251 31848 leader_election.cc:343] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.619869 31923 raft_consensus.cc:357] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.620535 31923 raft_consensus.cc:383] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.620770 31923 raft_consensus.cc:738] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.621364 31923 consensus_queue.cc:260] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.601356 31727 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:17.623737 31923 ts_tablet_manager.cc:1428] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.087s user 0.005s sys 0.001s
I20250831 07:58:17.630801 31925 ts_tablet_manager.cc:1428] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.074s user 0.005s sys 0.001s
I20250831 07:58:17.631913 31925 tablet_bootstrap.cc:492] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:17.636245 31923 tablet_bootstrap.cc:492] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:17.642975 32236 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:17.643946 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:17.644987 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:17.647435 31924 tablet_bootstrap.cc:492] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:17.648161 31924 ts_tablet_manager.cc:1397] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.189s user 0.018s sys 0.027s
I20250831 07:58:17.647480 31734 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:17.652529 31848 leader_election.cc:343] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.654371 31628 catalog_manager.cc:5582] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:17.664381 31734 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:17.670789 32223 consensus_queue.cc:1035] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:17.671546 31849 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:17.672827 31954 raft_consensus.cc:2802] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:17.673266 31954 raft_consensus.cc:695] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:17.673949 31954 consensus_queue.cc:237] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.673288 31629 catalog_manager.cc:5582] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:17.678866 31924 raft_consensus.cc:357] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.679548 31924 raft_consensus.cc:383] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.679795 31924 raft_consensus.cc:738] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.684434 31924 consensus_queue.cc:260] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.687076 31925 tablet_bootstrap.cc:654] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.705914 31924 ts_tablet_manager.cc:1428] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.057s user 0.005s sys 0.000s
I20250831 07:58:17.707157 31924 tablet_bootstrap.cc:492] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:17.683494 31923 tablet_bootstrap.cc:654] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.712956 31727 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:17.713621 32236 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:17.714421 31849 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:17.714241 32236 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.715634 32229 raft_consensus.cc:2802] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:17.716172 32229 raft_consensus.cc:695] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:17.716890 32236 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:17.717051 32229 consensus_queue.cc:237] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.718382 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:17.721101 31848 leader_election.cc:343] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.754832 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:17.762476 31849 leader_election.cc:343] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.763868 31849 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:17.764993 31954 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:17.766866 32223 consensus_queue.cc:1035] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:17.779628 31924 tablet_bootstrap.cc:654] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.786417 31923 tablet_bootstrap.cc:492] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:17.787002 31923 ts_tablet_manager.cc:1397] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.151s user 0.014s sys 0.004s
I20250831 07:58:17.787686 31628 catalog_manager.cc:5582] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:17.789171 31923 raft_consensus.cc:357] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.789695 31923 raft_consensus.cc:383] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.791496 31923 raft_consensus.cc:738] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.789628 31628 catalog_manager.cc:5582] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:17.792075 31923 consensus_queue.cc:260] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.794510 31923 ts_tablet_manager.cc:1428] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250831 07:58:17.794859 31925 tablet_bootstrap.cc:492] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:17.795336 31925 ts_tablet_manager.cc:1397] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.164s user 0.008s sys 0.022s
I20250831 07:58:17.795491 31923 tablet_bootstrap.cc:492] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:17.797883 31925 raft_consensus.cc:357] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.798593 31925 raft_consensus.cc:383] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.798555 31954 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:17.799006 31925 raft_consensus.cc:738] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.799139 31954 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.799675 31925 consensus_queue.cc:260] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.801405 31954 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:17.802491 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:17.802613 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:17.803673 31925 ts_tablet_manager.cc:1428] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.008s user 0.003s sys 0.006s
W20250831 07:58:17.804170 31848 leader_election.cc:343] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.804697 31925 tablet_bootstrap.cc:492] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
W20250831 07:58:17.806968 31849 leader_election.cc:343] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.807380 31849 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:17.808676 32229 raft_consensus.cc:2747] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:17.813427 31925 tablet_bootstrap.cc:654] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.814244 32221 consensus_queue.cc:1035] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:17.818439 31923 tablet_bootstrap.cc:654] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.818440 31925 tablet_bootstrap.cc:492] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:17.819273 31925 ts_tablet_manager.cc:1397] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20250831 07:58:17.822238 31925 raft_consensus.cc:357] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.823110 31925 raft_consensus.cc:383] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.823441 31925 raft_consensus.cc:738] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.824232 31925 consensus_queue.cc:260] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.827018 31925 ts_tablet_manager.cc:1428] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250831 07:58:17.828446 31925 tablet_bootstrap.cc:492] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:17.837872 31925 tablet_bootstrap.cc:654] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.844591 32230 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250831 07:58:17.852144 32221 consensus_queue.cc:1035] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:17.869820 31925 tablet_bootstrap.cc:492] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:17.870339 31925 ts_tablet_manager.cc:1397] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.002s
I20250831 07:58:17.882115 32220 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:17.891939 31924 tablet_bootstrap.cc:492] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:17.892475 31924 ts_tablet_manager.cc:1397] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.186s user 0.014s sys 0.007s
I20250831 07:58:17.895216 31924 raft_consensus.cc:357] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.888732 31925 raft_consensus.cc:357] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.902361 31925 raft_consensus.cc:383] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.902782 31925 raft_consensus.cc:738] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.902366 31924 raft_consensus.cc:383] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.903659 31924 raft_consensus.cc:738] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.904942 31924 consensus_queue.cc:260] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.907879 31924 ts_tablet_manager.cc:1428] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20250831 07:58:17.908828 31924 tablet_bootstrap.cc:492] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:17.912077 31925 consensus_queue.cc:260] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.913731 31925 ts_tablet_manager.cc:1428] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.028s user 0.004s sys 0.000s
I20250831 07:58:17.918174 31925 tablet_bootstrap.cc:492] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:17.923023 31924 tablet_bootstrap.cc:654] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.928308 31925 tablet_bootstrap.cc:654] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.932291 31924 tablet_bootstrap.cc:492] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:17.932520 31923 tablet_bootstrap.cc:492] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:17.932998 31923 ts_tablet_manager.cc:1397] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.138s user 0.012s sys 0.000s
I20250831 07:58:17.936323 31923 raft_consensus.cc:357] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.936950 31923 raft_consensus.cc:383] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.937202 31923 raft_consensus.cc:738] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.937777 31923 consensus_queue.cc:260] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.940042 31923 ts_tablet_manager.cc:1428] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250831 07:58:17.940981 31923 tablet_bootstrap.cc:492] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:17.946471 31923 tablet_bootstrap.cc:654] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.932734 31924 ts_tablet_manager.cc:1397] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20250831 07:58:17.950125 31924 raft_consensus.cc:357] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.950603 31924 raft_consensus.cc:383] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.950814 31924 raft_consensus.cc:738] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.951292 31924 consensus_queue.cc:260] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.953231 31924 ts_tablet_manager.cc:1428] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250831 07:58:17.954092 31924 tablet_bootstrap.cc:492] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:17.960240 31924 tablet_bootstrap.cc:654] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.965767 32230 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:17.966185 31923 tablet_bootstrap.cc:492] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:17.966840 31924 tablet_bootstrap.cc:492] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:17.967089 31923 ts_tablet_manager.cc:1397] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.003s
I20250831 07:58:17.966357 32230 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.967646 31924 ts_tablet_manager.cc:1397] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20250831 07:58:17.969628 31923 raft_consensus.cc:357] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.970144 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:17.970268 31923 raft_consensus.cc:383] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.970260 31924 raft_consensus.cc:357] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.970805 31923 raft_consensus.cc:738] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.972100 31924 raft_consensus.cc:383] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:17.972325 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:17.972517 31924 raft_consensus.cc:738] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:17.972508 31923 consensus_queue.cc:260] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.973450 31924 consensus_queue.cc:260] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:17.975112 31923 ts_tablet_manager.cc:1428] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250831 07:58:17.970872 31874 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:17.975890 31924 ts_tablet_manager.cc:1428] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250831 07:58:17.976287 31923 tablet_bootstrap.cc:492] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:17.976915 31924 tablet_bootstrap.cc:492] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:17.976768 31698 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:17.978317 32220 raft_consensus.cc:2802] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:17.978683 32220 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:17.979033 32220 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:17.984135 31699 leader_election.cc:343] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:17.984416 31924 tablet_bootstrap.cc:654] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.985781 32230 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:17.986857 31923 tablet_bootstrap.cc:654] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:17.995505 32220 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.001787 31925 tablet_bootstrap.cc:492] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:18.002262 31925 ts_tablet_manager.cc:1397] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.084s user 0.008s sys 0.000s
I20250831 07:58:18.004015 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:18.004508 32220 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.005594 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:18.006119 31874 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:18.011554 31699 leader_election.cc:343] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:18.013278 31924 tablet_bootstrap.cc:492] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:18.014878 31924 ts_tablet_manager.cc:1397] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.038s user 0.017s sys 0.003s
I20250831 07:58:18.013796 32220 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:18.016708 32220 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.004676 31925 raft_consensus.cc:357] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.018270 31925 raft_consensus.cc:383] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:18.018554 31925 raft_consensus.cc:738] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:18.018735 32220 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.019179 31925 consensus_queue.cc:260] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.019767 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:18.020243 31874 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:18.020923 31924 raft_consensus.cc:357] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:18.021437 31698 leader_election.cc:343] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:18.021862 31924 raft_consensus.cc:383] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:18.022197 31924 raft_consensus.cc:738] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:18.022166 31698 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:18.023126 31924 consensus_queue.cc:260] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.025310 31924 ts_tablet_manager.cc:1428] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.010s user 0.004s sys 0.001s
I20250831 07:58:18.023478 32220 raft_consensus.cc:2802] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:18.025946 32220 raft_consensus.cc:695] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:18.026506 31924 tablet_bootstrap.cc:492] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:18.032881 31925 ts_tablet_manager.cc:1428] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20250831 07:58:18.032996 31924 tablet_bootstrap.cc:654] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:18.028939 32220 consensus_queue.cc:237] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.043928 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:18.051908 32216 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:18.052531 32216 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.055429 31925 tablet_bootstrap.cc:492] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:18.055835 32216 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.056420 31806 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:18.056733 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:18.057276 31923 tablet_bootstrap.cc:492] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:18.059995 31923 ts_tablet_manager.cc:1397] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.084s user 0.015s sys 0.007s
W20250831 07:58:18.059662 31775 leader_election.cc:343] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:18.062996 31923 raft_consensus.cc:357] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.063657 31923 raft_consensus.cc:383] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:18.063948 31923 raft_consensus.cc:738] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:18.064605 31925 tablet_bootstrap.cc:654] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:18.064543 31923 consensus_queue.cc:260] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.066818 31923 ts_tablet_manager.cc:1428] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250831 07:58:18.071539 31699 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:18.072830 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:18.073378 31924 tablet_bootstrap.cc:492] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:18.073518 31874 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:18.073838 31924 ts_tablet_manager.cc:1397] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.008s
I20250831 07:58:18.074694 31773 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:18.075870 32231 raft_consensus.cc:2802] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:18.075975 32216 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:18.076269 32231 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:18.076414 32216 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:18.076522 31923 tablet_bootstrap.cc:492] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:18.076663 32231 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:18.076789 32216 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:18.078583 31924 raft_consensus.cc:357] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.079291 31924 raft_consensus.cc:383] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:18.079541 31924 raft_consensus.cc:738] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:18.080152 31924 consensus_queue.cc:260] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.081285 32218 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:18.161916 31923 tablet_bootstrap.cc:654] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:18.162462 32231 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.166656 32230 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:18.188463 32230 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.190412 32230 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.192209 32231 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.172854 31925 tablet_bootstrap.cc:492] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:18.193544 31925 ts_tablet_manager.cc:1397] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.138s user 0.015s sys 0.001s
I20250831 07:58:18.181797 32218 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.187057 32223 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:18.187727 32229 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:18.204159 31924 ts_tablet_manager.cc:1428] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.130s user 0.008s sys 0.005s
W20250831 07:58:18.207172 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 122840us
I20250831 07:58:18.207970 32324 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:18.208429 32324 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.210348 32324 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.197638 32223 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.214367 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:18.216168 31773 leader_election.cc:343] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:18.196130 32216 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.224332 31924 tablet_bootstrap.cc:492] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:18.225822 32216 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.199155 32229 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.228711 32223 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.243760 31923 tablet_bootstrap.cc:492] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:18.247121 31923 ts_tablet_manager.cc:1397] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.171s user 0.029s sys 0.020s
I20250831 07:58:18.252090 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:18.253823 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:18.254407 31733 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:18.242487 32068 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:18.263635 31773 leader_election.cc:343] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:18.408310 32229 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:18.408599 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 145031us
I20250831 07:58:18.428936 32068 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.300274 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:58:18.432044 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 46884.6us
I20250831 07:58:18.301975 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:18.301815 32013 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:18.491582 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 49865.8us
I20250831 07:58:18.440551 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:18.249078 32218 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.490653 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:18.521319 32068 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.300875 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:18.255473 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:18.522513 31874 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:18.523025 32221 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:18.501021 31875 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:18.524694 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:18.501533 32013 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.544098 32013 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.523530 32221 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.547428 32221 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:18.556610 31924 tablet_bootstrap.cc:700] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c: Time spent opening tablet: real 0.157s user 0.002s sys 0.022s
I20250831 07:58:18.564620 31923 raft_consensus.cc:357] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.565385 31923 raft_consensus.cc:383] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:18.565709 31923 raft_consensus.cc:738] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:18.567034 31924 tablet_bootstrap.cc:654] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:18.569554 31923 consensus_queue.cc:260] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.568410 32323 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:18.590821 31848 consensus_peers.cc:489] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 -> Peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Couldn't send request to peer 7512ba2ed3a94982a22a75f51a647e65. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:18.586526 31925 raft_consensus.cc:357] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.591871 31925 raft_consensus.cc:383] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:18.592286 31925 raft_consensus.cc:738] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:18.593068 31925 consensus_queue.cc:260] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.595585 31923 ts_tablet_manager.cc:1428] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.348s user 0.002s sys 0.022s
I20250831 07:58:18.590862 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:18.602783 31809 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:18.605839 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:18.612576 31680 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 07:58:18.578256 32323 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.618567 31923 tablet_bootstrap.cc:492] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:18.625805 31775 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:18.625964 31874 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
W20250831 07:58:18.626891 31775 leader_election.cc:343] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:18.627207 31775 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:18.627300 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:18.628346 32329 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:18.628399 32324 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:18.628788 32329 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:18.629158 32329 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:18.621491 32323 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:18.630934 31775 leader_election.cc:343] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:18.632261 31925 ts_tablet_manager.cc:1428] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.438s user 0.018s sys 0.011s
W20250831 07:58:18.639963 31698 leader_election.cc:343] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:18.640589 31698 leader_election.cc:343] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:18.641175 31698 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:18.642556 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:18.643850 31775 leader_election.cc:343] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:18.646561 31954 consensus_queue.cc:1035] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:18.646471 32013 raft_consensus.cc:2802] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:18.641203 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:18.649228 32013 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:18.653712 32013 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:18.656569 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:18.657194 31802 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:18.659353 32329 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.660430 32013 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.660806 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:18.661643 32329 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.662322 32013 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.646610 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:18.667197 31809 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:18.675792 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:18.675961 31699 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:18.676469 31805 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:18.677711 32328 raft_consensus.cc:2802] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:18.678138 32328 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:18.678527 32328 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:18.685746 31925 tablet_bootstrap.cc:492] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:18.690121 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:18.690688 31733 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:18.703090 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 102243us
I20250831 07:58:18.694541 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:18.706879 31802 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:18.708552 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 63560.1us
W20250831 07:58:18.707521 31848 leader_election.cc:343] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:18.711601 31805 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:18.708035 31631 catalog_manager.cc:5582] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:18.708310 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:18.717235 31732 raft_consensus.cc:2391] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 1: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:18.724992 31848 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:18.725185 31733 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:18.731298 31925 tablet_bootstrap.cc:654] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:18.733074 31775 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:18.735018 32326 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
W20250831 07:58:18.736384 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 49467.7us
W20250831 07:58:18.737054 31699 consensus_peers.cc:489] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c -> Peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Couldn't send request to peer 7512ba2ed3a94982a22a75f51a647e65. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:18.735620 32326 raft_consensus.cc:695] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:18.738430 31699 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:18.735967 32334 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:18.738852 32236 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:18.744607 32219 raft_consensus.cc:2802] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:18.745272 32219 raft_consensus.cc:695] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:18.746346 32219 consensus_queue.cc:237] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.747164 32321 consensus_queue.cc:1035] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250831 07:58:18.743216 32334 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:18.752315 32334 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:18.742736 32236 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.757025 32236 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:18.759281 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:18.941773 31923 tablet_bootstrap.cc:654] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:18.749423 32326 consensus_queue.cc:237] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.785915 31924 tablet_bootstrap.cc:492] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:18.901444 31698 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:18.944816 31924 ts_tablet_manager.cc:1397] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.728s user 0.011s sys 0.049s
I20250831 07:58:18.945048 32223 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:18.766350 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:18.776754 31773 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:18.948521 32323 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:18.948997 32323 raft_consensus.cc:695] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:58:18.949292 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 47988.5us
I20250831 07:58:18.949797 32323 consensus_queue.cc:237] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:18.950985 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 174359us
W20250831 07:58:18.854666 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 72442.5us
I20250831 07:58:18.784133 32328 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.958957 32328 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:18.960723 31802 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
W20250831 07:58:18.936905 31849 leader_election.cc:343] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:18.961396 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:18.962949 31849 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:18.988067 31805 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:18.992276 31699 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:18.947427 31924 raft_consensus.cc:357] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:18.995770 31924 raft_consensus.cc:383] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:18.996101 31924 raft_consensus.cc:738] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:18.996636 31924 consensus_queue.cc:260] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.003415 31954 raft_consensus.cc:2747] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:19.002282 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:19.007688 31923 tablet_bootstrap.cc:492] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:19.008184 31923 ts_tablet_manager.cc:1397] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.394s user 0.023s sys 0.024s
I20250831 07:58:19.009876 31773 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:19.010694 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 67665.2us
W20250831 07:58:19.011721 31849 leader_election.cc:343] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
W20250831 07:58:19.012641 31698 leader_election.cc:343] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:19.016268 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:19.016876 31803 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:19.017401 32334 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.033519 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:19.052352 31923 raft_consensus.cc:357] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.054390 31923 raft_consensus.cc:383] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:19.041472 32068 raft_consensus.cc:2747] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:19.054622 31923 raft_consensus.cc:738] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:19.055980 31923 consensus_queue.cc:260] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.058722 32336 raft_consensus.cc:2802] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:19.168816 31923 ts_tablet_manager.cc:1428] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.160s user 0.006s sys 0.000s
I20250831 07:58:19.055184 31876 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:19.188580 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:19.189932 31725 raft_consensus.cc:2391] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 1: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:19.056306 32236 consensus_queue.cc:1035] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.123s
I20250831 07:58:19.181586 31881 raft_consensus.cc:2391] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 1: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:19.069336 32336 raft_consensus.cc:695] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:58:19.214675 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 195345us
I20250831 07:58:19.222903 31923 tablet_bootstrap.cc:492] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
W20250831 07:58:19.227252 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 178388us
W20250831 07:58:19.231550 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 199149us
I20250831 07:58:19.233273 31925 tablet_bootstrap.cc:492] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:19.233911 31925 ts_tablet_manager.cc:1397] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.548s user 0.034s sys 0.072s
I20250831 07:58:19.242607 32349 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:19.243263 32349 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.258440 32349 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:19.383801 31924 ts_tablet_manager.cc:1428] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.439s user 0.009s sys 0.005s
W20250831 07:58:19.400507 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 168397us
I20250831 07:58:19.297703 32334 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:19.428294 31924 tablet_bootstrap.cc:492] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:19.448868 32221 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:19.430706 31803 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:19.479024 32349 consensus_queue.cc:1035] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20250831 07:58:19.488592 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 206323us
I20250831 07:58:19.299571 32336 consensus_queue.cc:237] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.490141 31876 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:19.490186 32361 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:19.476975 32221 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.500669 32356 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:19.503703 31925 raft_consensus.cc:357] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.505018 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:19.507773 31925 raft_consensus.cc:383] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:19.508113 31925 raft_consensus.cc:738] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:19.508687 31925 consensus_queue.cc:260] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.511636 32356 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.511986 32361 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.516264 31849 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:19.517432 32334 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:19.518836 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:19.519464 31803 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:19.525442 31925 ts_tablet_manager.cc:1428] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.291s user 0.008s sys 0.000s
I20250831 07:58:19.532130 32356 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:19.533067 32350 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:19.623972 31879 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:19.629933 32221 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:19.624842 32350 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.625516 32361 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:19.728816 32329 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:19.750303 32329 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.752599 32329 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:19.753381 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:19.757381 32350 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:19.758044 32349 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:19.758581 32349 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:19.762614 32349 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:19.764854 32223 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:19.765408 32223 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.767393 32223 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:19.778228 31925 tablet_bootstrap.cc:492] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:19.778652 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:19.779362 31725 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:19.782225 31924 tablet_bootstrap.cc:654] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:19.781028 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:19.784776 32230 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:19.788427 32230 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:19.790475 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 43967.6us
I20250831 07:58:19.793784 32230 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:19.789664 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:19.810923 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 253) took 1035 ms. Trace:
I20250831 07:58:19.812845 31699 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:19.811463 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:19.815979 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 178) took 1076 ms. Trace:
W20250831 07:58:19.817173 31775 leader_election.cc:343] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:19.816429 31881 rpcz_store.cc:276] 0831 07:58:18.739636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:18.740585 (+ 949us) service_pool.cc:225] Handling call
0831 07:58:19.815738 (+1075153us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b5400247a54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:19.815898 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":267292416}
I20250831 07:58:19.822181 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:19.822793 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:19.820390 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:19.827863 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 245) took 1098 ms. Trace:
I20250831 07:58:19.828691 31699 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:19.814818 32328 raft_consensus.cc:2802] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:19.831770 32328 raft_consensus.cc:695] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:19.823789 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:58:19.833320 31698 consensus_peers.cc:489] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c -> Peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Couldn't send request to peer 8247a38b550649aa9cd6e7668dcaec10. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:19.833537 31883 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:19.817202 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:19.828258 31875 rpcz_store.cc:276] 0831 07:58:18.728969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:18.813190 (+ 84221us) service_pool.cc:225] Handling call
0831 07:58:19.827205 (+1014015us) spinlock_profiling.cc:243] Waited 41.4 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:58:19.827375 (+ 170us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:19.827523 (+ 148us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:19.827549 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:19.827594 (+ 45us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:19.827623 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:19.827675 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:19.827771 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":95270656}
I20250831 07:58:19.848104 32356 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:19.848672 32356 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.850505 32356 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:19.853421 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:19.858383 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:19.860077 31848 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.812s user 0.010s sys 0.016s
W20250831 07:58:19.862285 31848 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.815s user 0.011s sys 0.016s
W20250831 07:58:19.862612 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 814974us
I20250831 07:58:19.867933 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:19.871567 31773 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:19.874507 31726 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:19.874384 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:19.811429 31804 rpcz_store.cc:276] 0831 07:58:18.775862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:18.786968 (+ 11106us) service_pool.cc:225] Handling call
0831 07:58:19.809980 (+1023012us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b5800245198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:58:19.810170 (+ 190us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:19.810347 (+ 177us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:19.810386 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:19.810433 (+ 47us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:19.810485 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:19.810566 (+ 81us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:19.810707 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7816960}
I20250831 07:58:19.838591 32328 consensus_queue.cc:237] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.871183 31809 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:19.892830 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:19.896103 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:19.898375 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:19.901155 31883 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:19.902429 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 248) took 1035 ms. Trace:
I20250831 07:58:19.902740 31883 rpcz_store.cc:276] 0831 07:58:18.866669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:18.867187 (+ 518us) service_pool.cc:225] Handling call
0831 07:58:19.823541 (+956354us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:58:19.902374 (+ 78833us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25778944}
I20250831 07:58:19.906646 31925 tablet_bootstrap.cc:654] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:19.916766 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:19.919916 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:19.921594 31631 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59664 (request call id 108) took 1333 ms. Trace:
W20250831 07:58:19.921998 31680 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 07:58:19.927678 32363 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:19.928241 32363 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:19.929705 32218 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:19.944592 31923 tablet_bootstrap.cc:654] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:19.951272 31628 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59666 (request call id 125) took 1234 ms. Trace:
I20250831 07:58:19.957968 31773 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:19.951774 31628 rpcz_store.cc:276] 0831 07:58:18.719356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:18.719585 (+ 229us) service_pool.cc:225] Handling call
0831 07:58:19.951190 (+1231605us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":92572,"reported_tablets":1}
I20250831 07:58:19.921952 31631 rpcz_store.cc:276] 0831 07:58:18.589343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:18.589833 (+ 490us) service_pool.cc:225] Handling call
0831 07:58:19.921513 (+1331680us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:58:18.998977 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:58:19.001828 (+ 2851us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:58:19.001858 (+ 30us) write_op.cc:435] Acquired schema lock
0831 07:58:19.001874 (+ 16us) tablet.cc:660] Decoding operations
0831 07:58:19.002193 (+ 319us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:58:19.002278 (+ 85us) write_op.cc:641] Partition lock acquired for write op
0831 07:58:19.002302 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0831 07:58:19.005047 (+ 2745us) tablet.cc:699] Row locks acquired
0831 07:58:19.005064 (+ 17us) write_op.cc:260] PREPARE: finished
0831 07:58:19.005182 (+ 118us) write_op.cc:270] Start()
0831 07:58:19.005282 (+ 100us) write_op.cc:276] Timestamp: P: 1756627099005147 usec, L: 0
0831 07:58:19.005319 (+ 37us) op_driver.cc:348] REPLICATION: starting
0831 07:58:19.005947 (+ 628us) log.cc:844] Serialized 537 byte log entry
0831 07:58:19.242907 (+236960us) op_driver.cc:464] REPLICATION: finished
0831 07:58:19.406290 (+163383us) write_op.cc:301] APPLY: starting
0831 07:58:19.406363 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0831 07:58:19.406449 (+ 86us) tablet.cc:1368] finished BulkCheckPresence
0831 07:58:19.406556 (+ 107us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:58:19.406796 (+ 240us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:58:19.406953 (+ 157us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0831 07:58:19.406996 (+ 43us) write_op.cc:312] APPLY: finished
0831 07:58:19.429828 (+ 22832us) log.cc:844] Serialized 22 byte log entry
0831 07:58:19.439475 (+ 9647us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:19.446120 (+ 6645us) write_op.cc:454] Released schema lock
0831 07:58:19.450063 (+ 3943us) write_op.cc:341] FINISH: Updating metrics
0831 07:58:19.500111 (+ 50048us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:19.500161 (+ 50us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":163133,"apply.run_cpu_time_us":2191,"apply.run_wall_time_us":77068,"num_ops":1,"prepare.queue_time_us":27456,"prepare.run_cpu_time_us":18187,"prepare.run_wall_time_us":269019,"raft.queue_time_us":20607,"raft.run_cpu_time_us":13303,"raft.run_wall_time_us":47302,"replication_time_us":237422,"thread_start_us":107901,"threads_started":4,"wal-append.queue_time_us":172368}]]}
W20250831 07:58:19.966984 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 61623.9us
I20250831 07:58:19.983659 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:19.985466 31924 tablet_bootstrap.cc:492] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:19.986132 31924 ts_tablet_manager.cc:1397] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.558s user 0.016s sys 0.022s
I20250831 07:58:19.990926 31725 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:19.991694 32231 raft_consensus.cc:2762] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250831 07:58:19.998301 31925 tablet_bootstrap.cc:492] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:19.998819 31925 ts_tablet_manager.cc:1397] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.221s user 0.005s sys 0.033s
I20250831 07:58:19.969478 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:19.968614 32326 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:19.960002 32218 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.019838 31924 raft_consensus.cc:357] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.020879 32223 consensus_queue.cc:1035] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:20.022572 31924 raft_consensus.cc:383] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:19.974711 32363 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:20.022933 31924 raft_consensus.cc:738] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:20.025161 31924 consensus_queue.cc:260] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.026075 31848 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:19.970214 32361 consensus_queue.cc:1035] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20250831 07:58:20.032953 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:20.033494 31849 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:20.035382 32229 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:20.035866 32229 raft_consensus.cc:695] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:20.036695 32229 consensus_queue.cc:237] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.037444 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:20.039433 31848 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:20.050186 32344 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:20.018354 31877 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 0.
I20250831 07:58:20.053010 32218 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:20.053639 32370 log.cc:927] Time spent T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20250831 07:58:20.079592 31848 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:20.094659 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 63320.2us
I20250831 07:58:20.102787 31773 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:20.104509 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:20.104863 32363 raft_consensus.cc:2747] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:20.107810 31924 ts_tablet_manager.cc:1428] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.121s user 0.008s sys 0.000s
I20250831 07:58:20.118309 31924 tablet_bootstrap.cc:492] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:20.118842 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 246) took 1263 ms. Trace:
I20250831 07:58:20.119213 31879 rpcz_store.cc:276] 0831 07:58:18.855131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:18.861032 (+ 5901us) service_pool.cc:225] Handling call
0831 07:58:19.534373 (+673341us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:58:19.623642 (+ 89269us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:19.778549 (+154907us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:19.778581 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:58:19.779021 (+ 440us) log.cc:844] Serialized 26 byte log entry
0831 07:58:19.909180 (+130159us) spinlock_profiling.cc:243] Waited 5.11 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce507e39c 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdce7c3b2bb
0831 07:58:19.933354 (+ 24174us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:20.009148 (+ 75794us) log.cc:844] Serialized 14 byte log entry
0831 07:58:20.097681 (+ 88533us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:20.118633 (+ 20952us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:58:20.118669 (+ 36us) raft_consensus.cc:1683] finished
0831 07:58:20.118676 (+ 7us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:20.118781 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65818,"prepare.queue_time_us":81012,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":63778,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":5957,"spinlock_wait_cycles":79501056,"thread_start_us":186412,"threads_started":2,"wal-append.queue_time_us":155971}
I20250831 07:58:20.114645 32329 raft_consensus.cc:2802] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:20.133898 32351 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:20.134488 32351 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.130445 32329 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:20.152151 32329 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:20.172555 31925 raft_consensus.cc:357] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.173589 31925 raft_consensus.cc:383] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:20.173934 31925 raft_consensus.cc:738] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:20.174602 31924 tablet_bootstrap.cc:654] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:20.186439 32364 raft_consensus.cc:2747] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:20.194058 32351 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:20.203271 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:20.205232 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:20.211925 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 249) took 1163 ms. Trace:
I20250831 07:58:20.224681 31809 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:20.226075 31699 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:20.225970 31925 consensus_queue.cc:260] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.225200 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:20.234503 32366 raft_consensus.cc:2747] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:20.239074 31699 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:20.212479 31726 rpcz_store.cc:276] 0831 07:58:19.025003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:19.033211 (+ 8208us) service_pool.cc:225] Handling call
0831 07:58:19.874176 (+840965us) spinlock_profiling.cc:243] Waited 944 us on lock 0x7b5800000f98. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:58:19.874358 (+ 182us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:19.949513 (+ 75155us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:19.949566 (+ 53us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:58:19.950065 (+ 499us) log.cc:844] Serialized 26 byte log entry
0831 07:58:20.017533 (+ 67468us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:20.074802 (+ 57269us) log.cc:844] Serialized 14 byte log entry
0831 07:58:20.160542 (+ 85740us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:20.165463 (+ 4921us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:58:20.168250 (+ 2787us) raft_consensus.cc:1683] finished
0831 07:58:20.171171 (+ 2921us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:20.187488 (+ 16317us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29925,"prepare.queue_time_us":42773,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":48691,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":3428,"spinlock_wait_cycles":2170624,"thread_start_us":54436,"threads_started":1,"wal-append.queue_time_us":168554}
I20250831 07:58:20.242138 31954 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:20.242596 31954 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.249732 32326 consensus_queue.cc:1035] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:20.256181 31954 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:20.256911 32327 raft_consensus.cc:2747] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:20.260380 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:20.237741 32356 raft_consensus.cc:2747] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:20.266175 31699 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:20.274991 31925 ts_tablet_manager.cc:1428] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.276s user 0.014s sys 0.004s
I20250831 07:58:20.250061 32231 consensus_queue.cc:1035] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20250831 07:58:20.275993 31699 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:58:20.278118 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 44435.2us
I20250831 07:58:20.290494 32359 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:20.291008 32359 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.291882 32346 raft_consensus.cc:2747] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:20.292589 32329 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.292925 32359 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:20.294450 32329 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:20.298322 31699 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:20.298954 31775 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:20.300755 31925 tablet_bootstrap.cc:492] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65: Bootstrap starting.
I20250831 07:58:20.284106 32336 raft_consensus.cc:2747] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:20.316236 31631 catalog_manager.cc:5582] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:20.317909 31631 catalog_manager.cc:5582] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:20.309304 32329 raft_consensus.cc:2747] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:20.312994 32328 raft_consensus.cc:2747] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:20.340404 32360 consensus_queue.cc:1035] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.056s
I20250831 07:58:20.345331 31699 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:20.329448 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:20.354414 31775 consensus_peers.cc:489] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 -> Peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Couldn't send request to peer a0d845d01c8c4894895f66e3beae298c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:20.315891 32327 consensus_queue.cc:1035] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20250831 07:58:20.356736 32220 raft_consensus.cc:2747] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:20.354102 31876 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 0.
I20250831 07:58:20.368779 31698 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:20.374823 31925 tablet_bootstrap.cc:654] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:20.355577 32359 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:20.380960 31923 tablet_bootstrap.cc:492] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:20.381626 31923 ts_tablet_manager.cc:1397] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 1.159s user 0.023s sys 0.072s
I20250831 07:58:20.384418 32367 raft_consensus.cc:2802] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:20.384841 32367 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:20.385160 32367 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:20.387367 31924 tablet_bootstrap.cc:492] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:20.388003 31924 ts_tablet_manager.cc:1397] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.270s user 0.015s sys 0.015s
I20250831 07:58:20.382862 32359 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.405826 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:20.406476 31879 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:20.450848 32347 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:20.458758 31924 raft_consensus.cc:357] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.459441 31924 raft_consensus.cc:383] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:20.459654 31924 raft_consensus.cc:738] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:20.457690 32359 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:20.468003 31924 consensus_queue.cc:260] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.470124 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:20.454943 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:20.451406 32347 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:20.501609 32394 log.cc:927] Time spent T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20250831 07:58:20.502146 31924 ts_tablet_manager.cc:1428] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.114s user 0.006s sys 0.000s
I20250831 07:58:20.504793 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:20.499549 31879 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:20.499603 32347 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:20.509109 31923 raft_consensus.cc:357] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.520387 31923 raft_consensus.cc:383] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:20.521277 31773 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:20.526526 31924 tablet_bootstrap.cc:492] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c: Bootstrap starting.
I20250831 07:58:20.520797 31923 raft_consensus.cc:738] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:20.527398 31923 consensus_queue.cc:260] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.530912 31923 ts_tablet_manager.cc:1428] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.149s user 0.009s sys 0.010s
I20250831 07:58:20.533660 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:20.509786 32367 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.537058 31808 raft_consensus.cc:2391] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:20.537554 31727 raft_consensus.cc:2391] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 1: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:20.538491 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:20.540307 32360 raft_consensus.cc:2802] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 07:58:20.540788 32360 raft_consensus.cc:695] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:20.540962 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:20.541592 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:20.541800 31877 raft_consensus.cc:2391] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:20.541724 32360 consensus_queue.cc:237] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.555150 31699 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:20.558861 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:20.545001 32367 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:20.556277 31849 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:20.559919 32327 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:20.564857 31954 raft_consensus.cc:2747] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:20.550768 32343 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:20.566618 31699 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:20.570506 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:20.643910 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 77363.8us
I20250831 07:58:20.644452 31773 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:20.569209 32343 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.657371 32356 raft_consensus.cc:2747] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250831 07:58:20.658704 32216 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:20.664637 32346 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:20.678741 31924 tablet_bootstrap.cc:654] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:20.683621 31801 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:20.689724 31925 tablet_bootstrap.cc:492] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65: No bootstrap required, opened a new log
I20250831 07:58:20.690215 31925 ts_tablet_manager.cc:1397] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65: Time spent bootstrapping tablet: real 0.401s user 0.014s sys 0.011s
I20250831 07:58:20.691181 31954 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:20.695158 32236 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:20.681993 32346 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.705374 32331 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:20.708933 32343 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:20.709160 32331 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.714543 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:20.720611 32346 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:20.721768 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:20.721771 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:20.723625 31775 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:20.725801 32343 raft_consensus.cc:2747] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:20.728590 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:20.729306 31806 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 0.
I20250831 07:58:20.730471 31848 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:20.728591 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:20.733652 32335 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 07:58:20.734061 32335 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:20.732641 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:20.734508 32335 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:20.735461 31849 leader_election.cc:343] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:20.725723 32331 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:20.738236 31699 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:20.745270 31923 tablet_bootstrap.cc:492] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:20.764832 32223 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:20.770272 31925 raft_consensus.cc:357] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.770985 31925 raft_consensus.cc:383] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:20.771241 31925 raft_consensus.cc:738] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Initialized, Role: FOLLOWER
I20250831 07:58:20.772493 31925 consensus_queue.cc:260] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.782053 31925 ts_tablet_manager.cc:1428] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65: Time spent starting tablet: real 0.092s user 0.003s sys 0.002s
I20250831 07:58:20.784118 31923 tablet_bootstrap.cc:654] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:20.790807 32223 consensus_queue.cc:1035] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250831 07:58:20.801694 32335 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:20.815979 31698 consensus_peers.cc:489] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c -> Peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Couldn't send request to peer 8247a38b550649aa9cd6e7668dcaec10. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:20.820566 31628 catalog_manager.cc:5582] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:20.822748 31628 catalog_manager.cc:5582] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:20.824658 31628 catalog_manager.cc:5582] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:20.830489 31727 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:20.846783 31630 catalog_manager.cc:5582] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:20.862527 32335 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:20.869480 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:20.875509 31923 tablet_bootstrap.cc:492] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:20.879297 31923 ts_tablet_manager.cc:1397] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.134s user 0.003s sys 0.016s
I20250831 07:58:20.874214 31924 tablet_bootstrap.cc:492] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c: No bootstrap required, opened a new log
I20250831 07:58:20.884264 31924 ts_tablet_manager.cc:1397] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c: Time spent bootstrapping tablet: real 0.368s user 0.018s sys 0.013s
I20250831 07:58:20.886709 31923 raft_consensus.cc:357] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.889355 31923 raft_consensus.cc:383] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:20.889803 31923 raft_consensus.cc:738] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:20.890561 31923 consensus_queue.cc:260] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.895560 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:20.908221 31802 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:20.897294 31849 leader_election.cc:343] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Illegal state: must be running to vote when last-logged opid is not known
I20250831 07:58:20.913425 31630 catalog_manager.cc:5582] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:20.915465 31802 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:20.916600 31848 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:20.917596 32331 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader election won for term 1
W20250831 07:58:20.917793 31773 consensus_peers.cc:489] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 -> Peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Couldn't send request to peer 8247a38b550649aa9cd6e7668dcaec10. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20250831 07:58:20.918066 32331 raft_consensus.cc:695] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:20.918884 32331 consensus_queue.cc:237] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.927050 31923 ts_tablet_manager.cc:1428] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.047s user 0.004s sys 0.002s
I20250831 07:58:20.928007 31923 tablet_bootstrap.cc:492] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:20.931566 31924 raft_consensus.cc:357] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.932284 31924 raft_consensus.cc:383] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:20.932543 31924 raft_consensus.cc:738] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Initialized, Role: FOLLOWER
I20250831 07:58:20.933102 31924 consensus_queue.cc:260] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.934530 31923 tablet_bootstrap.cc:654] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:20.938879 32361 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:20.939421 32361 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:20.941378 32361 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:20.942534 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:20.943004 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:20.944634 31773 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:20.947170 32361 raft_consensus.cc:2747] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:20.948505 31924 ts_tablet_manager.cc:1428] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20250831 07:58:20.973712 31631 catalog_manager.cc:5582] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:21.012650 32361 consensus_queue.cc:1035] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:21.018114 31923 tablet_bootstrap.cc:492] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:21.018605 31923 ts_tablet_manager.cc:1397] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.003s
I20250831 07:58:21.021034 31923 raft_consensus.cc:357] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:21.022161 31923 raft_consensus.cc:383] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:21.022395 31923 raft_consensus.cc:738] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:21.023067 31923 consensus_queue.cc:260] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:21.023622 32328 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:21.024931 31923 ts_tablet_manager.cc:1428] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250831 07:58:21.025211 32328 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:21.027263 32328 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:21.028402 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:21.030761 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:21.031502 31801 raft_consensus.cc:2391] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 1: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:21.032660 31699 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:21.033866 32328 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:21.037608 31923 tablet_bootstrap.cc:492] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10: Bootstrap starting.
I20250831 07:58:21.046809 32068 consensus_queue.cc:1035] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 07:58:21.058320 32328 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:21.090306 31923 tablet_bootstrap.cc:654] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10: Neither blocks nor log segments found. Creating new log.
I20250831 07:58:21.105053 31801 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:21.126492 31923 tablet_bootstrap.cc:492] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10: No bootstrap required, opened a new log
I20250831 07:58:21.127041 31923 ts_tablet_manager.cc:1397] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10: Time spent bootstrapping tablet: real 0.090s user 0.011s sys 0.001s
I20250831 07:58:21.129712 31923 raft_consensus.cc:357] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:21.130424 31923 raft_consensus.cc:383] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 07:58:21.130699 31923 raft_consensus.cc:738] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Initialized, Role: FOLLOWER
I20250831 07:58:21.131912 31923 consensus_queue.cc:260] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:21.134598 31923 ts_tablet_manager.cc:1428] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250831 07:58:21.190179 32333 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:21.191476 32333 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:21.195147 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:21.197432 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:21.195147 32333 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:21.200188 31848 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:21.202030 32331 raft_consensus.cc:2747] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:21.308780 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 07:58:21.317242 32419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:58:21.320725 32331 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:21.323266 32333 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 07:58:21.329537 31880 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
W20250831 07:58:21.336920 32420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:58:21.337292 32374 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:21.340416 32374 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:21.361094 32374 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:21.361203 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:21.364022 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:21.373489 31848 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:21.382819 32333 raft_consensus.cc:2747] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:21.434537 31876 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:21.495199 30564 server_base.cc:1047] running on GCE node
W20250831 07:58:21.505725 32426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 07:58:21.506829 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 07:58:21.507071 30564 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 07:58:21.507258 30564 hybrid_clock.cc:648] HybridClock initialized: now 1756627101507241 us; error 0 us; skew 500 ppm
I20250831 07:58:21.507861 30564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 07:58:21.511108 30564 webserver.cc:480] Webserver started at http://127.29.217.4:43093/ using document root <none> and password file <none>
I20250831 07:58:21.511751 30564 fs_manager.cc:362] Metadata directory not provided
I20250831 07:58:21.512054 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 07:58:21.512413 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 07:58:21.513963 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-3-root/instance:
uuid: "7ecb257174554a26b2c35ad04b0eabae"
format_stamp: "Formatted at 2025-08-31 07:58:21 on dist-test-slave-87tg"
I20250831 07:58:21.519809 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20250831 07:58:21.520188 32331 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:21.520660 32331 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:21.522579 32331 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:21.524155 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:21.524639 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:21.526012 31848 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:21.526393 32432 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 07:58:21.527053 32331 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 07:58:21.527927 30564 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20250831 07:58:21.528301 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-3-root
uuid: "7ecb257174554a26b2c35ad04b0eabae"
format_stamp: "Formatted at 2025-08-31 07:58:21 on dist-test-slave-87tg"
I20250831 07:58:21.528656 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 07:58:21.541353 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 07:58:21.542750 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 07:58:21.544896 30564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 07:58:21.548280 30564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 07:58:21.548552 30564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 07:58:21.548878 30564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 07:58:21.549086 30564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 07:58:21.564234 31876 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 0 FOLLOWER]: Advancing to term 1
I20250831 07:58:21.710362 30564 rpc_server.cc:307] RPC server started. Bound to: 127.29.217.4:43319
I20250831 07:58:21.710453 32495 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.217.4:43319 every 8 connection(s)
I20250831 07:58:21.809969 32496 heartbeater.cc:344] Connected to a master server at 127.29.217.62:40065
I20250831 07:58:21.810667 32496 heartbeater.cc:461] Registering TS with master...
I20250831 07:58:21.811733 32496 heartbeater.cc:507] Master 127.29.217.62:40065 requested a full tablet report, sending...
I20250831 07:58:21.819590 31629 ts_manager.cc:194] Registered new tserver with Master: 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:21.821533 31629 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43950
I20250831 07:58:21.829387 30564 auto_leader_rebalancer-test.cc:417] add a tserver: 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:58:22.156396 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 46053.5us
I20250831 07:58:22.221830 31801 consensus_queue.cc:237] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:22.231968 31876 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:22.234074 31734 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:22.233969 32361 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:22.239995 32324 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:58:22.245280 31773 consensus_peers.cc:489] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 719b7e1bd5624bb9a08892e34c5e1b47. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:22.259344 32361 raft_consensus.cc:2953] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.264454 31875 raft_consensus.cc:2953] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.264921 31733 raft_consensus.cc:2953] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.284951 31628 catalog_manager.cc:5582] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.301070 31734 consensus_queue.cc:237] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:22.314828 31802 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:22.316519 32223 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:22.318280 31875 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:22.326504 32223 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:58:22.337802 31698 consensus_peers.cc:489] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b32e93c114645eeb4e707778c91e12e. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:22.346671 32223 raft_consensus.cc:2953] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.349174 31801 raft_consensus.cc:2953] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.353046 31875 raft_consensus.cc:2953] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.384095 31631 catalog_manager.cc:5582] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.401942 31734 consensus_queue.cc:237] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
W20250831 07:58:22.427731 31698 consensus_peers.cc:489] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca7b8b0d72c94ed386b217b936955022. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:22.430090 31875 raft_consensus.cc:1273] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:22.431576 32223 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:22.431701 31802 raft_consensus.cc:1273] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:22.435290 32223 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:22.470407 32500 raft_consensus.cc:2953] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.472390 31875 raft_consensus.cc:2953] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.485209 31801 raft_consensus.cc:2953] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.502310 31628 catalog_manager.cc:5582] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.505949 31802 consensus_queue.cc:237] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:22.521365 31875 raft_consensus.cc:1273] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250831 07:58:22.523890 31773 consensus_peers.cc:489] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2e336f43934f118ecacdc1e1d2da60. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:22.525110 31732 raft_consensus.cc:1273] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:22.526422 32425 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:22.527372 32361 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:22.541325 32324 raft_consensus.cc:2953] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.542459 31875 raft_consensus.cc:2953] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.579908 31628 catalog_manager.cc:5582] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.581840 31732 raft_consensus.cc:2953] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:22.758586 32526 ts_tablet_manager.cc:927] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:22.764993 32526 tablet_copy_client.cc:323] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.2:37261
I20250831 07:58:22.804860 31819 tablet_copy_service.cc:140] P 7512ba2ed3a94982a22a75f51a647e65: Received BeginTabletCopySession request for tablet 719b7e1bd5624bb9a08892e34c5e1b47 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43428)
I20250831 07:58:22.805557 31819 tablet_copy_service.cc:161] P 7512ba2ed3a94982a22a75f51a647e65: Beginning new tablet copy session on tablet 719b7e1bd5624bb9a08892e34c5e1b47 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43428: session id = 7ecb257174554a26b2c35ad04b0eabae-719b7e1bd5624bb9a08892e34c5e1b47
I20250831 07:58:22.817945 31819 tablet_copy_source_session.cc:215] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:22.821756 32526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719b7e1bd5624bb9a08892e34c5e1b47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:22.824425 32496 heartbeater.cc:499] Master 127.29.217.62:40065 was elected leader, sending a full tablet report...
I20250831 07:58:22.845865 32526 tablet_copy_client.cc:806] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:22.846557 32526 tablet_copy_client.cc:670] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:22.851742 32526 tablet_copy_client.cc:538] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:22.870131 32526 tablet_bootstrap.cc:492] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:22.875572 32530 ts_tablet_manager.cc:927] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:58:22.889427 32530 tablet_copy_client.cc:323] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.1:45027
I20250831 07:58:22.899042 32532 ts_tablet_manager.cc:927] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:58:22.916656 32532 tablet_copy_client.cc:323] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.1:45027
I20250831 07:58:22.936164 31744 tablet_copy_service.cc:140] P a0d845d01c8c4894895f66e3beae298c: Received BeginTabletCopySession request for tablet 2b32e93c114645eeb4e707778c91e12e from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43230)
I20250831 07:58:22.936754 31744 tablet_copy_service.cc:161] P a0d845d01c8c4894895f66e3beae298c: Beginning new tablet copy session on tablet 2b32e93c114645eeb4e707778c91e12e from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43230: session id = 7ecb257174554a26b2c35ad04b0eabae-2b32e93c114645eeb4e707778c91e12e
I20250831 07:58:22.941193 32526 tablet_bootstrap.cc:492] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:22.942204 32526 tablet_bootstrap.cc:492] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:22.942858 32526 ts_tablet_manager.cc:1397] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.073s user 0.020s sys 0.003s
I20250831 07:58:22.945156 31743 tablet_copy_service.cc:140] P a0d845d01c8c4894895f66e3beae298c: Received BeginTabletCopySession request for tablet ca7b8b0d72c94ed386b217b936955022 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43230)
I20250831 07:58:22.945768 31743 tablet_copy_service.cc:161] P a0d845d01c8c4894895f66e3beae298c: Beginning new tablet copy session on tablet ca7b8b0d72c94ed386b217b936955022 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43230: session id = 7ecb257174554a26b2c35ad04b0eabae-ca7b8b0d72c94ed386b217b936955022
I20250831 07:58:22.947152 32526 raft_consensus.cc:357] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:22.948011 32526 raft_consensus.cc:738] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:22.948661 32526 consensus_queue.cc:260] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:22.954115 32526 ts_tablet_manager.cc:1428] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20250831 07:58:22.955714 31819 tablet_copy_service.cc:342] P 7512ba2ed3a94982a22a75f51a647e65: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-719b7e1bd5624bb9a08892e34c5e1b47 received from {username='slave'} at 127.0.0.1:43428
I20250831 07:58:22.956247 31819 tablet_copy_service.cc:434] P 7512ba2ed3a94982a22a75f51a647e65: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-719b7e1bd5624bb9a08892e34c5e1b47 on tablet 719b7e1bd5624bb9a08892e34c5e1b47 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:22.964643 31743 tablet_copy_source_session.cc:215] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:22.965952 31744 tablet_copy_source_session.cc:215] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:22.969738 32532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca7b8b0d72c94ed386b217b936955022. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:22.972854 32526 ts_tablet_manager.cc:927] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:22.999289 32526 tablet_copy_client.cc:323] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.2:37261
I20250831 07:58:23.000994 31819 tablet_copy_service.cc:140] P 7512ba2ed3a94982a22a75f51a647e65: Received BeginTabletCopySession request for tablet 6b2e336f43934f118ecacdc1e1d2da60 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43428)
I20250831 07:58:23.001528 31819 tablet_copy_service.cc:161] P 7512ba2ed3a94982a22a75f51a647e65: Beginning new tablet copy session on tablet 6b2e336f43934f118ecacdc1e1d2da60 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43428: session id = 7ecb257174554a26b2c35ad04b0eabae-6b2e336f43934f118ecacdc1e1d2da60
I20250831 07:58:23.001860 32532 tablet_copy_client.cc:806] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:23.002497 32532 tablet_copy_client.cc:670] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:23.011313 31819 tablet_copy_source_session.cc:215] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:23.020310 32532 tablet_copy_client.cc:538] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:23.025400 32530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b32e93c114645eeb4e707778c91e12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:23.041487 32532 tablet_bootstrap.cc:492] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:23.042958 32526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b2e336f43934f118ecacdc1e1d2da60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:23.044571 32530 tablet_copy_client.cc:806] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:23.046283 32530 tablet_copy_client.cc:670] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:23.077503 32530 tablet_copy_client.cc:538] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:23.081441 32526 tablet_copy_client.cc:806] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:23.082046 32526 tablet_copy_client.cc:670] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:23.085769 32526 tablet_copy_client.cc:538] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:23.099031 32530 tablet_bootstrap.cc:492] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:23.105433 32526 tablet_bootstrap.cc:492] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:23.108439 32532 tablet_bootstrap.cc:492] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:23.114544 32532 tablet_bootstrap.cc:492] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:23.115284 32532 ts_tablet_manager.cc:1397] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.074s user 0.019s sys 0.003s
I20250831 07:58:23.123350 32532 raft_consensus.cc:357] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:23.124238 32532 raft_consensus.cc:738] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:23.124845 32532 consensus_queue.cc:260] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:23.128680 32532 ts_tablet_manager.cc:1428] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.013s user 0.002s sys 0.006s
I20250831 07:58:23.135946 31744 tablet_copy_service.cc:342] P a0d845d01c8c4894895f66e3beae298c: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-ca7b8b0d72c94ed386b217b936955022 received from {username='slave'} at 127.0.0.1:43230
I20250831 07:58:23.136396 31744 tablet_copy_service.cc:434] P a0d845d01c8c4894895f66e3beae298c: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-ca7b8b0d72c94ed386b217b936955022 on tablet ca7b8b0d72c94ed386b217b936955022 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:23.166504 32526 tablet_bootstrap.cc:492] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:23.167443 32530 tablet_bootstrap.cc:492] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:23.168488 32526 tablet_bootstrap.cc:492] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:23.169255 32526 ts_tablet_manager.cc:1397] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.064s user 0.024s sys 0.001s
I20250831 07:58:23.172144 32526 raft_consensus.cc:357] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:23.173074 32526 raft_consensus.cc:738] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:23.173768 32530 tablet_bootstrap.cc:492] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:23.173722 32526 consensus_queue.cc:260] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:23.175595 32530 ts_tablet_manager.cc:1397] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.077s user 0.017s sys 0.013s
I20250831 07:58:23.178403 32530 raft_consensus.cc:357] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:23.179236 32530 raft_consensus.cc:738] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:23.179860 32530 consensus_queue.cc:260] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:23.188023 32530 ts_tablet_manager.cc:1428] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.012s user 0.004s sys 0.005s
I20250831 07:58:23.191094 32526 ts_tablet_manager.cc:1428] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250831 07:58:23.193200 31744 tablet_copy_service.cc:342] P a0d845d01c8c4894895f66e3beae298c: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-2b32e93c114645eeb4e707778c91e12e received from {username='slave'} at 127.0.0.1:43230
I20250831 07:58:23.193622 31744 tablet_copy_service.cc:434] P a0d845d01c8c4894895f66e3beae298c: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-2b32e93c114645eeb4e707778c91e12e on tablet 2b32e93c114645eeb4e707778c91e12e with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:23.193708 31819 tablet_copy_service.cc:342] P 7512ba2ed3a94982a22a75f51a647e65: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-6b2e336f43934f118ecacdc1e1d2da60 received from {username='slave'} at 127.0.0.1:43428
I20250831 07:58:23.194176 31819 tablet_copy_service.cc:434] P 7512ba2ed3a94982a22a75f51a647e65: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-6b2e336f43934f118ecacdc1e1d2da60 on tablet 6b2e336f43934f118ecacdc1e1d2da60 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:23.238909 32471 raft_consensus.cc:1215] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:23.351902 32471 raft_consensus.cc:1215] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:23.383273 32471 raft_consensus.cc:1215] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:23.422703 32471 raft_consensus.cc:1215] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:23.665043 32361 raft_consensus.cc:1062] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:23.667043 32361 consensus_queue.cc:237] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:23.685333 31876 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:23.686353 32471 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:23.687461 32361 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:23.688347 32524 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:23.688647 31732 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:23.690044 32539 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:23.708503 32539 raft_consensus.cc:2953] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:23.746325 31734 raft_consensus.cc:2953] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:23.760110 31876 raft_consensus.cc:2953] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:23.763453 31629 catalog_manager.cc:5582] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:23.757685 32471 raft_consensus.cc:2953] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:23.826812 31808 consensus_queue.cc:237] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:23.831441 32471 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:23.832260 31876 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:23.834616 32547 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:23.836499 32547 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:23.848124 32524 raft_consensus.cc:2953] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:23.851910 31876 raft_consensus.cc:2953] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:23.856506 32471 raft_consensus.cc:2953] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:23.858278 31617 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 719b7e1bd5624bb9a08892e34c5e1b47 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:23.867781 31631 catalog_manager.cc:5582] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:23.874325 32517 raft_consensus.cc:1062] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:23.879401 32517 consensus_queue.cc:237] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:23.886972 31716 tablet_service.cc:1515] Processing DeleteTablet for tablet 719b7e1bd5624bb9a08892e34c5e1b47 with delete_type TABLET_DATA_TOMBSTONED (TS a0d845d01c8c4894895f66e3beae298c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57666
I20250831 07:58:23.900164 32553 tablet_replica.cc:331] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c: stopping tablet replica
I20250831 07:58:23.901229 32553 raft_consensus.cc:2241] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:23.901914 32553 raft_consensus.cc:2270] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:23.924232 32553 ts_tablet_manager.cc:1905] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:23.924894 32471 raft_consensus.cc:1273] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:23.926843 31802 raft_consensus.cc:1273] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:23.927652 32327 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:23.928143 31880 raft_consensus.cc:1273] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:23.928755 32500 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:23.932788 32547 raft_consensus.cc:1062] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:23.936275 32500 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:23.936022 32547 consensus_queue.cc:237] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:23.944728 32553 ts_tablet_manager.cc:1918] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:23.947019 32553 log.cc:1199] T 719b7e1bd5624bb9a08892e34c5e1b47 P a0d845d01c8c4894895f66e3beae298c: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-0-root/wals/719b7e1bd5624bb9a08892e34c5e1b47
I20250831 07:58:23.948758 31618 catalog_manager.cc:4928] TS a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): tablet 719b7e1bd5624bb9a08892e34c5e1b47 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:23.955791 31880 raft_consensus.cc:1273] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:23.958072 31734 raft_consensus.cc:1273] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:23.958669 32547 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:23.959875 32470 raft_consensus.cc:1273] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:23.960137 32361 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:23.963163 32361 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:23.989308 32556 raft_consensus.cc:1062] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:23.991701 32556 consensus_queue.cc:237] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:23.994446 32554 raft_consensus.cc:2953] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.004233 31802 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:24.005635 32517 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:24.024605 31876 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:24.026487 32470 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:24.030001 31876 raft_consensus.cc:2953] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.041574 32500 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:24.052107 31801 raft_consensus.cc:2953] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.044703 32500 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20250831 07:58:24.090623 31631 catalog_manager.cc:5582] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.102488 31734 raft_consensus.cc:2953] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.116791 32547 raft_consensus.cc:2953] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.122519 31880 raft_consensus.cc:2953] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.127329 32470 raft_consensus.cc:2953] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.164022 32554 raft_consensus.cc:2953] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.168431 32470 raft_consensus.cc:2953] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.179669 32469 raft_consensus.cc:2953] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.180123 31734 consensus_queue.cc:237] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:24.199869 31628 catalog_manager.cc:5582] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.203011 32470 raft_consensus.cc:1273] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:24.210347 31875 raft_consensus.cc:1273] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:24.213646 32567 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:24.216884 31876 raft_consensus.cc:2953] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.223460 31801 raft_consensus.cc:2953] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.224645 32554 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:58:24.234788 32327 raft_consensus.cc:2953] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.239702 31801 consensus_queue.cc:237] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:24.244822 31876 raft_consensus.cc:2953] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.250671 31618 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet ca7b8b0d72c94ed386b217b936955022 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:24.256177 31629 catalog_manager.cc:5582] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.267225 32469 raft_consensus.cc:2953] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.277854 32470 raft_consensus.cc:1273] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:24.279345 32569 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:58:24.279795 31876 raft_consensus.cc:1273] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:24.287330 32361 raft_consensus.cc:2953] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.288787 32470 raft_consensus.cc:2953] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.291987 32524 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:24.291036 31631 catalog_manager.cc:5582] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:24.298161 31876 raft_consensus.cc:2953] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.313489 31617 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 6b2e336f43934f118ecacdc1e1d2da60 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:24.321702 31727 consensus_queue.cc:237] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:24.323951 31628 catalog_manager.cc:5582] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:24.325672 31791 tablet_service.cc:1515] Processing DeleteTablet for tablet ca7b8b0d72c94ed386b217b936955022 with delete_type TABLET_DATA_TOMBSTONED (TS 7512ba2ed3a94982a22a75f51a647e65 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39068
I20250831 07:58:24.404649 32571 tablet_replica.cc:331] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65: stopping tablet replica
I20250831 07:58:24.405656 32571 raft_consensus.cc:2241] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:24.406348 32571 raft_consensus.cc:2270] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:24.407544 32469 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:24.413307 32500 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:24.422912 31801 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:24.431326 32327 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:24.443586 32566 raft_consensus.cc:2953] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.455040 31801 raft_consensus.cc:2953] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.459168 31716 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b2e336f43934f118ecacdc1e1d2da60 with delete_type TABLET_DATA_TOMBSTONED (TS a0d845d01c8c4894895f66e3beae298c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57666
I20250831 07:58:24.466964 32470 raft_consensus.cc:2953] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.470611 31618 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 2b32e93c114645eeb4e707778c91e12e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:24.472501 32571 ts_tablet_manager.cc:1905] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:24.474701 31629 catalog_manager.cc:5582] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:24.492568 32573 tablet_replica.cc:331] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c: stopping tablet replica
I20250831 07:58:24.493798 32573 raft_consensus.cc:2241] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:24.494557 32573 raft_consensus.cc:2270] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:24.495476 31873 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b32e93c114645eeb4e707778c91e12e with delete_type TABLET_DATA_TOMBSTONED (TS 8247a38b550649aa9cd6e7668dcaec10 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57332
I20250831 07:58:24.501607 32573 ts_tablet_manager.cc:1905] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:24.510474 32574 tablet_replica.cc:331] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10: stopping tablet replica
I20250831 07:58:24.511493 32574 raft_consensus.cc:2241] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:24.530759 32574 raft_consensus.cc:2270] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:24.533823 32571 ts_tablet_manager.cc:1918] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:24.534338 32571 log.cc:1199] T ca7b8b0d72c94ed386b217b936955022 P 7512ba2ed3a94982a22a75f51a647e65: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-1-root/wals/ca7b8b0d72c94ed386b217b936955022
I20250831 07:58:24.536950 31617 catalog_manager.cc:4928] TS 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): tablet ca7b8b0d72c94ed386b217b936955022 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:24.537691 32574 ts_tablet_manager.cc:1905] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:24.540526 32573 ts_tablet_manager.cc:1918] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:24.540949 32573 log.cc:1199] T 6b2e336f43934f118ecacdc1e1d2da60 P a0d845d01c8c4894895f66e3beae298c: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-0-root/wals/6b2e336f43934f118ecacdc1e1d2da60
I20250831 07:58:24.542991 31618 catalog_manager.cc:4928] TS a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): tablet 6b2e336f43934f118ecacdc1e1d2da60 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:24.558652 32574 ts_tablet_manager.cc:1918] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:24.559183 32574 log.cc:1199] T 2b32e93c114645eeb4e707778c91e12e P 8247a38b550649aa9cd6e7668dcaec10: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-2-root/wals/2b32e93c114645eeb4e707778c91e12e
I20250831 07:58:24.560921 31616 catalog_manager.cc:4928] TS 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): tablet 2b32e93c114645eeb4e707778c91e12e (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:25.566480 31733 consensus_queue.cc:237] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:25.581576 31808 raft_consensus.cc:1273] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:25.587128 31877 raft_consensus.cc:1273] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:25.590849 32578 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:25.595149 32577 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:58:25.597795 31698 consensus_peers.cc:489] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d510d241da34d25a50f2d65d17041a0. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:25.613348 32578 raft_consensus.cc:2953] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:25.619060 31880 raft_consensus.cc:2953] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:25.621758 31808 raft_consensus.cc:2953] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:25.636288 31733 consensus_queue.cc:237] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:25.634011 31629 catalog_manager.cc:5582] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:25.657312 31801 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:25.658180 31880 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:25.660562 32578 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:58:25.661511 31698 consensus_peers.cc:489] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a62a36682136474ca552fbc53190e09e. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:25.662019 32566 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:25.678588 32578 raft_consensus.cc:2953] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:25.684103 31802 raft_consensus.cc:2953] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:25.689584 31880 raft_consensus.cc:2953] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:25.699954 31801 consensus_queue.cc:237] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:25.709367 31877 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:25.709972 31631 catalog_manager.cc:5582] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:25.711030 32569 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:58:25.718331 31773 consensus_peers.cc:489] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e12d332258d4d0b8d0421b49ed5d0c0. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:25.724906 31733 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:25.728780 32569 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250831 07:58:25.742280 32569 raft_consensus.cc:2953] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:25.743758 31733 raft_consensus.cc:2953] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:25.770306 31631 catalog_manager.cc:5582] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:25.784987 31877 raft_consensus.cc:2953] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:25.810969 31877 consensus_queue.cc:237] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:25.819008 31802 raft_consensus.cc:1273] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:25.820601 31733 raft_consensus.cc:1273] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:25.822295 32333 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:25.823318 32572 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:58:25.842097 31847 consensus_peers.cc:489] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f319de3ac3c4b9a9b87cd497e02c51c. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:25.846459 32572 raft_consensus.cc:2953] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:25.853164 31802 raft_consensus.cc:2953] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:25.853440 31733 raft_consensus.cc:2953] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:25.879889 31631 catalog_manager.cc:5582] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:26.237221 32608 ts_tablet_manager.cc:927] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:26.293293 32608 tablet_copy_client.cc:323] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.3:42585
W20250831 07:58:26.317665 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 48724.9us
I20250831 07:58:26.312358 32611 ts_tablet_manager.cc:927] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:58:26.433182 32615 ts_tablet_manager.cc:927] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
W20250831 07:58:26.449115 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 168769us
I20250831 07:58:26.437853 32611 tablet_copy_client.cc:323] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.1:45027
I20250831 07:58:26.453871 32615 tablet_copy_client.cc:323] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.1:45027
I20250831 07:58:26.469029 31744 tablet_copy_service.cc:140] P a0d845d01c8c4894895f66e3beae298c: Received BeginTabletCopySession request for tablet 9d510d241da34d25a50f2d65d17041a0 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43230)
I20250831 07:58:26.469584 31744 tablet_copy_service.cc:161] P a0d845d01c8c4894895f66e3beae298c: Beginning new tablet copy session on tablet 9d510d241da34d25a50f2d65d17041a0 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43230: session id = 7ecb257174554a26b2c35ad04b0eabae-9d510d241da34d25a50f2d65d17041a0
I20250831 07:58:26.472028 31743 tablet_copy_service.cc:140] P a0d845d01c8c4894895f66e3beae298c: Received BeginTabletCopySession request for tablet a62a36682136474ca552fbc53190e09e from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43230)
I20250831 07:58:26.472568 32619 ts_tablet_manager.cc:927] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:26.472582 31743 tablet_copy_service.cc:161] P a0d845d01c8c4894895f66e3beae298c: Beginning new tablet copy session on tablet a62a36682136474ca552fbc53190e09e from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43230: session id = 7ecb257174554a26b2c35ad04b0eabae-a62a36682136474ca552fbc53190e09e
I20250831 07:58:26.474331 32619 tablet_copy_client.cc:323] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.2:37261
I20250831 07:58:26.475885 31819 tablet_copy_service.cc:140] P 7512ba2ed3a94982a22a75f51a647e65: Received BeginTabletCopySession request for tablet 9e12d332258d4d0b8d0421b49ed5d0c0 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43428)
I20250831 07:58:26.476387 31819 tablet_copy_service.cc:161] P 7512ba2ed3a94982a22a75f51a647e65: Beginning new tablet copy session on tablet 9e12d332258d4d0b8d0421b49ed5d0c0 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43428: session id = 7ecb257174554a26b2c35ad04b0eabae-9e12d332258d4d0b8d0421b49ed5d0c0
I20250831 07:58:26.485828 31819 tablet_copy_source_session.cc:215] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:26.490950 32619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e12d332258d4d0b8d0421b49ed5d0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:26.494176 31744 tablet_copy_source_session.cc:215] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:26.494756 31743 tablet_copy_source_session.cc:215] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:26.506748 31893 tablet_copy_service.cc:140] P 8247a38b550649aa9cd6e7668dcaec10: Received BeginTabletCopySession request for tablet 4f319de3ac3c4b9a9b87cd497e02c51c from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:51246)
I20250831 07:58:26.516865 31893 tablet_copy_service.cc:161] P 8247a38b550649aa9cd6e7668dcaec10: Beginning new tablet copy session on tablet 4f319de3ac3c4b9a9b87cd497e02c51c from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:51246: session id = 7ecb257174554a26b2c35ad04b0eabae-4f319de3ac3c4b9a9b87cd497e02c51c
I20250831 07:58:26.527354 31893 tablet_copy_source_session.cc:215] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:26.536013 32611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d510d241da34d25a50f2d65d17041a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:26.543800 32608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f319de3ac3c4b9a9b87cd497e02c51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:26.548862 32615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a62a36682136474ca552fbc53190e09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:26.572702 32619 tablet_copy_client.cc:806] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:26.582144 32619 tablet_copy_client.cc:670] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:26.599138 32619 tablet_copy_client.cc:538] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:26.616520 32619 tablet_bootstrap.cc:492] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:26.648095 32619 tablet_bootstrap.cc:492] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:26.649070 32619 tablet_bootstrap.cc:492] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:26.649698 32619 ts_tablet_manager.cc:1397] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.009s
I20250831 07:58:26.652524 32619 raft_consensus.cc:357] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:26.653409 32619 raft_consensus.cc:738] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:26.654088 32619 consensus_queue.cc:260] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:26.663900 32619 ts_tablet_manager.cc:1428] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20250831 07:58:26.665418 31819 tablet_copy_service.cc:342] P 7512ba2ed3a94982a22a75f51a647e65: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-9e12d332258d4d0b8d0421b49ed5d0c0 received from {username='slave'} at 127.0.0.1:43428
I20250831 07:58:26.665859 31819 tablet_copy_service.cc:434] P 7512ba2ed3a94982a22a75f51a647e65: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-9e12d332258d4d0b8d0421b49ed5d0c0 on tablet 9e12d332258d4d0b8d0421b49ed5d0c0 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:26.669970 32611 tablet_copy_client.cc:806] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:26.675851 32611 tablet_copy_client.cc:670] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:26.681939 32611 tablet_copy_client.cc:538] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:26.690855 32611 tablet_bootstrap.cc:492] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:26.692335 32608 tablet_copy_client.cc:806] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:26.700332 32608 tablet_copy_client.cc:670] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:26.664023 32615 tablet_copy_client.cc:806] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:26.701188 32615 tablet_copy_client.cc:670] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:26.706063 32615 tablet_copy_client.cc:538] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:26.707451 32608 tablet_copy_client.cc:538] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:26.717674 32608 tablet_bootstrap.cc:492] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:26.721732 32615 tablet_bootstrap.cc:492] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:26.738509 32611 tablet_bootstrap.cc:492] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:26.740190 32611 tablet_bootstrap.cc:492] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:26.740980 32611 ts_tablet_manager.cc:1397] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.050s user 0.025s sys 0.000s
I20250831 07:58:26.744035 32611 raft_consensus.cc:357] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:26.744961 32611 raft_consensus.cc:738] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:26.745645 32611 consensus_queue.cc:260] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:26.749413 32611 ts_tablet_manager.cc:1428] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250831 07:58:26.751220 31744 tablet_copy_service.cc:342] P a0d845d01c8c4894895f66e3beae298c: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-9d510d241da34d25a50f2d65d17041a0 received from {username='slave'} at 127.0.0.1:43230
I20250831 07:58:26.751659 31744 tablet_copy_service.cc:434] P a0d845d01c8c4894895f66e3beae298c: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-9d510d241da34d25a50f2d65d17041a0 on tablet 9d510d241da34d25a50f2d65d17041a0 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:26.778885 32470 raft_consensus.cc:1215] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:26.785776 32608 tablet_bootstrap.cc:492] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:26.793215 32608 tablet_bootstrap.cc:492] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:26.793740 32615 tablet_bootstrap.cc:492] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:26.794001 32608 ts_tablet_manager.cc:1397] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.077s user 0.020s sys 0.005s
I20250831 07:58:26.794734 32615 tablet_bootstrap.cc:492] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:26.795451 32615 ts_tablet_manager.cc:1397] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.074s user 0.022s sys 0.000s
I20250831 07:58:26.796893 32608 raft_consensus.cc:357] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:26.797748 32608 raft_consensus.cc:738] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:26.798362 32608 consensus_queue.cc:260] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:26.801918 32615 raft_consensus.cc:357] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:26.802735 32615 raft_consensus.cc:738] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:26.803481 32615 consensus_queue.cc:260] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:26.809712 32608 ts_tablet_manager.cc:1428] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.015s user 0.001s sys 0.005s
I20250831 07:58:26.815722 31893 tablet_copy_service.cc:342] P 8247a38b550649aa9cd6e7668dcaec10: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-4f319de3ac3c4b9a9b87cd497e02c51c received from {username='slave'} at 127.0.0.1:51246
I20250831 07:58:26.816241 31893 tablet_copy_service.cc:434] P 8247a38b550649aa9cd6e7668dcaec10: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-4f319de3ac3c4b9a9b87cd497e02c51c on tablet 4f319de3ac3c4b9a9b87cd497e02c51c with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:26.823092 32615 ts_tablet_manager.cc:1428] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20250831 07:58:26.839928 31744 tablet_copy_service.cc:342] P a0d845d01c8c4894895f66e3beae298c: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-a62a36682136474ca552fbc53190e09e received from {username='slave'} at 127.0.0.1:43230
I20250831 07:58:26.840404 31744 tablet_copy_service.cc:434] P a0d845d01c8c4894895f66e3beae298c: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-a62a36682136474ca552fbc53190e09e on tablet a62a36682136474ca552fbc53190e09e with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:27.047791 32470 raft_consensus.cc:1215] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20250831 07:58:27.190675 32492 debug-util.cc:398] Leaking SignalData structure 0x7b08002fb020 after lost signal to thread 30567
W20250831 07:58:27.192710 32492 debug-util.cc:398] Leaking SignalData structure 0x7b08002fb700 after lost signal to thread 31666
W20250831 07:58:27.194856 32492 debug-util.cc:398] Leaking SignalData structure 0x7b0800307fe0 after lost signal to thread 31758
W20250831 07:58:27.197731 32492 debug-util.cc:398] Leaking SignalData structure 0x7b0800132d20 after lost signal to thread 31833
W20250831 07:58:27.201802 32492 debug-util.cc:398] Leaking SignalData structure 0x7b0800255840 after lost signal to thread 32495
I20250831 07:58:27.229286 32470 raft_consensus.cc:1215] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:27.305958 32470 raft_consensus.cc:1215] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20250831 07:58:27.312274 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 75736.5us
I20250831 07:58:27.405043 32612 raft_consensus.cc:1062] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:27.408596 32612 consensus_queue.cc:237] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:27.428900 32470 raft_consensus.cc:1273] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:27.429032 31801 raft_consensus.cc:1273] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:27.430716 31875 raft_consensus.cc:1273] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:27.432578 32628 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:27.433535 32629 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:27.438390 32627 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:27.484508 32627 raft_consensus.cc:2953] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:27.493212 32470 raft_consensus.cc:2953] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:27.506803 31801 raft_consensus.cc:2953] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:27.530730 31875 raft_consensus.cc:2953] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:27.566576 31631 catalog_manager.cc:5582] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:27.642899 31728 consensus_queue.cc:237] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:27.650126 32470 raft_consensus.cc:1273] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:27.654536 32612 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:58:27.667976 32628 raft_consensus.cc:2953] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:27.669657 32589 raft_consensus.cc:1062] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:27.672405 32589 consensus_queue.cc:237] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:27.678968 31875 raft_consensus.cc:1273] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:27.678961 32469 raft_consensus.cc:2953] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:27.684609 32566 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:58:27.695034 31875 raft_consensus.cc:2953] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:27.702834 32470 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:27.709229 31728 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:27.712378 32630 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:27.714522 32630 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:27.723915 32602 raft_consensus.cc:1062] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:27.726959 32602 consensus_queue.cc:237] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:27.748589 31879 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:27.757661 32566 raft_consensus.cc:1062] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:27.761102 32566 consensus_queue.cc:237] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:27.762427 32569 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:27.769284 31618 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9d510d241da34d25a50f2d65d17041a0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:27.784634 32609 raft_consensus.cc:2953] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:27.802847 32470 raft_consensus.cc:2953] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:27.810990 32471 raft_consensus.cc:1273] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:27.815294 32471 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:27.814227 31734 raft_consensus.cc:2953] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:27.817245 32612 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:27.823311 31879 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:27.826597 32566 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:27.827021 32604 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250831 07:58:27.838155 31801 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:27.841646 31734 raft_consensus.cc:1273] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:27.842415 32627 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:27.864321 32599 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250831 07:58:27.865937 32642 log.cc:927] Time spent T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20250831 07:58:27.939528 31630 catalog_manager.cc:5582] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250831 07:58:27.952509 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 85995.5us
I20250831 07:58:27.952734 31629 catalog_manager.cc:5582] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250831 07:58:28.004990 32647 log.cc:927] Time spent T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
W20250831 07:58:28.028388 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 58326.9us
I20250831 07:58:28.036202 31791 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d510d241da34d25a50f2d65d17041a0 with delete_type TABLET_DATA_TOMBSTONED (TS 7512ba2ed3a94982a22a75f51a647e65 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39068
I20250831 07:58:28.041251 31801 consensus_queue.cc:237] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:28.055747 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 202747us
I20250831 07:58:28.061358 32471 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:28.063872 32657 tablet_replica.cc:331] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65: stopping tablet replica
I20250831 07:58:28.065248 32657 raft_consensus.cc:2241] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:28.070436 32609 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:28.077267 31728 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:28.078190 32657 raft_consensus.cc:2270] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:28.090512 32644 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250831 07:58:28.094952 32605 raft_consensus.cc:2953] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.093269 31801 raft_consensus.cc:1273] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:28.102366 32657 ts_tablet_manager.cc:1905] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:28.111897 31800 raft_consensus.cc:2953] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.148329 32569 raft_consensus.cc:2953] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.149951 32471 raft_consensus.cc:2953] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.184885 31728 raft_consensus.cc:2953] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.205893 32468 raft_consensus.cc:2953] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.213096 31630 catalog_manager.cc:5582] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
W20250831 07:58:28.217235 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 82494.9us
I20250831 07:58:28.222236 32599 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
W20250831 07:58:28.226289 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 86697.1us
I20250831 07:58:28.243155 31617 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9e12d332258d4d0b8d0421b49ed5d0c0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:28.228757 31882 raft_consensus.cc:2953] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.254128 32657 ts_tablet_manager.cc:1918] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:28.255627 32657 log.cc:1199] T 9d510d241da34d25a50f2d65d17041a0 P 7512ba2ed3a94982a22a75f51a647e65: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-1-root/wals/9d510d241da34d25a50f2d65d17041a0
W20250831 07:58:28.263089 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 137799us
I20250831 07:58:28.268263 31628 catalog_manager.cc:5582] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.271229 31617 catalog_manager.cc:4928] TS 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): tablet 9d510d241da34d25a50f2d65d17041a0 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:28.296761 31734 consensus_queue.cc:237] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:28.318774 31873 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e12d332258d4d0b8d0421b49ed5d0c0 with delete_type TABLET_DATA_TOMBSTONED (TS 8247a38b550649aa9cd6e7668dcaec10 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57332
I20250831 07:58:28.344698 31800 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:28.344866 32468 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:28.346642 32652 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:58:28.347656 32605 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:28.382493 32603 raft_consensus.cc:2953] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.383291 31730 raft_consensus.cc:2953] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.391330 32628 raft_consensus.cc:2953] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.397473 32468 raft_consensus.cc:2953] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.402379 31801 raft_consensus.cc:2953] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.409826 32667 tablet_replica.cc:331] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10: stopping tablet replica
I20250831 07:58:28.416445 32471 raft_consensus.cc:2953] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.420246 31808 raft_consensus.cc:2953] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.413826 32667 raft_consensus.cc:2241] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:28.422155 32667 pending_rounds.cc:70] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10: Trying to abort 1 pending ops.
I20250831 07:58:28.422428 32667 pending_rounds.cc:77] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7195144633019871232 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } } }
I20250831 07:58:28.437870 32667 raft_consensus.cc:2915] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20250831 07:58:28.438362 32667 raft_consensus.cc:2887] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250831 07:58:28.438715 32667 raft_consensus.cc:2270] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:28.439630 31630 catalog_manager.cc:5582] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.441751 31618 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a62a36682136474ca552fbc53190e09e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:28.451985 32667 ts_tablet_manager.cc:1905] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:28.469964 31873 tablet_service.cc:1515] Processing DeleteTablet for tablet a62a36682136474ca552fbc53190e09e with delete_type TABLET_DATA_TOMBSTONED (TS 8247a38b550649aa9cd6e7668dcaec10 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57332
I20250831 07:58:28.463708 31628 catalog_manager.cc:5582] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:28.513087 32667 ts_tablet_manager.cc:1918] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:28.514848 31877 consensus_queue.cc:237] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:28.518051 32667 log.cc:1199] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 8247a38b550649aa9cd6e7668dcaec10: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-2-root/wals/9e12d332258d4d0b8d0421b49ed5d0c0
I20250831 07:58:28.519924 31616 catalog_manager.cc:4928] TS 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): tablet 9e12d332258d4d0b8d0421b49ed5d0c0 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:28.520596 32471 raft_consensus.cc:1273] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:28.521948 32603 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:58:28.524293 32667 tablet_replica.cc:331] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10: stopping tablet replica
I20250831 07:58:28.525609 32667 raft_consensus.cc:2241] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:28.526630 32667 raft_consensus.cc:2270] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:28.531868 32654 raft_consensus.cc:2953] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.535336 32470 raft_consensus.cc:2953] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.549021 32667 ts_tablet_manager.cc:1905] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:28.553153 31616 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 4f319de3ac3c4b9a9b87cd497e02c51c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:28.549119 31725 raft_consensus.cc:1273] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:28.557349 31630 catalog_manager.cc:5582] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.564486 32654 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20250831 07:58:28.584089 31725 raft_consensus.cc:2953] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:28.597679 31791 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f319de3ac3c4b9a9b87cd497e02c51c with delete_type TABLET_DATA_TOMBSTONED (TS 7512ba2ed3a94982a22a75f51a647e65 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39068
I20250831 07:58:28.598872 32657 tablet_replica.cc:331] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65: stopping tablet replica
I20250831 07:58:28.621083 32657 raft_consensus.cc:2241] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:28.621964 32657 raft_consensus.cc:2270] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:28.629020 32667 ts_tablet_manager.cc:1918] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:28.632094 32667 log.cc:1199] T a62a36682136474ca552fbc53190e09e P 8247a38b550649aa9cd6e7668dcaec10: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-2-root/wals/a62a36682136474ca552fbc53190e09e
I20250831 07:58:28.630954 32657 ts_tablet_manager.cc:1905] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:28.633682 31616 catalog_manager.cc:4928] TS 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): tablet a62a36682136474ca552fbc53190e09e (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:28.664188 32657 ts_tablet_manager.cc:1918] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:28.664811 32657 log.cc:1199] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7512ba2ed3a94982a22a75f51a647e65: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-1-root/wals/4f319de3ac3c4b9a9b87cd497e02c51c
I20250831 07:58:28.674399 31617 catalog_manager.cc:4928] TS 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): tablet 4f319de3ac3c4b9a9b87cd497e02c51c (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:29.757200 31879 consensus_queue.cc:237] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:29.764099 31800 raft_consensus.cc:1273] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:29.765825 32665 consensus_queue.cc:1035] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:29.766557 31728 raft_consensus.cc:1273] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250831 07:58:29.766914 31847 consensus_peers.cc:489] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59e33bfa463944f89cfbb00d1247323a. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:29.767916 32682 consensus_queue.cc:1035] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:29.777683 32665 raft_consensus.cc:2953] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:29.786975 31800 raft_consensus.cc:2953] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:29.808823 31628 catalog_manager.cc:5582] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:29.811566 31879 consensus_queue.cc:237] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:29.813808 31728 raft_consensus.cc:2953] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
W20250831 07:58:29.828133 31847 consensus_peers.cc:489] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcbe3794f3fc40819ffbbfef02a071c3. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:29.829008 31800 raft_consensus.cc:1273] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:29.829113 31730 raft_consensus.cc:1273] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:29.830511 32665 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:29.836719 32665 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:29.861896 32681 raft_consensus.cc:2953] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:29.873394 31800 raft_consensus.cc:2953] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:29.906895 31730 raft_consensus.cc:2953] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:29.908149 31733 consensus_queue.cc:237] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:29.935494 31631 catalog_manager.cc:5582] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:29.950951 31883 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:29.952468 32662 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:58:29.953424 31698 consensus_peers.cc:489] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2acd7b3ac5f94f53ac1ef93264415686. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:29.955761 31800 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:29.957338 32662 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:29.968019 32662 raft_consensus.cc:2953] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:29.977018 31800 raft_consensus.cc:2953] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:29.983282 31733 consensus_queue.cc:237] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:29.988580 31628 catalog_manager.cc:5582] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:29.992213 31883 raft_consensus.cc:2953] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:30.006775 31802 raft_consensus.cc:1273] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:30.010071 32671 consensus_queue.cc:1035] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:58:30.019623 31698 consensus_peers.cc:489] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb2667d2ec934433ac10d395c1d29db4. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:30.021849 31879 raft_consensus.cc:1273] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:30.029579 32628 consensus_queue.cc:1035] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:30.060479 32662 raft_consensus.cc:2953] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:30.063939 31801 raft_consensus.cc:2953] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:30.063428 31879 raft_consensus.cc:2953] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:30.091121 31631 catalog_manager.cc:5582] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
W20250831 07:58:30.264426 31663 debug-util.cc:398] Leaking SignalData structure 0x7b0800204c20 after lost signal to thread 30567
W20250831 07:58:30.265309 31663 debug-util.cc:398] Leaking SignalData structure 0x7b0800205d20 after lost signal to thread 31666
W20250831 07:58:30.266372 31663 debug-util.cc:398] Leaking SignalData structure 0x7b080018e9c0 after lost signal to thread 31758
W20250831 07:58:30.267207 31663 debug-util.cc:398] Leaking SignalData structure 0x7b08002045e0 after lost signal to thread 31833
W20250831 07:58:30.268178 31663 debug-util.cc:398] Leaking SignalData structure 0x7b08002ce5c0 after lost signal to thread 31907
W20250831 07:58:30.269132 31663 debug-util.cc:398] Leaking SignalData structure 0x7b08002d55e0 after lost signal to thread 32495
W20250831 07:58:30.269516 31663 debug-util.cc:398] Leaking SignalData structure 0x7b080014d940 after lost signal to thread 32657
W20250831 07:58:30.270296 31663 debug-util.cc:398] Leaking SignalData structure 0x7b080014a920 after lost signal to thread 32667
I20250831 07:58:30.393821 32699 ts_tablet_manager.cc:927] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:30.420017 32701 ts_tablet_manager.cc:927] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:30.425123 32699 tablet_copy_client.cc:323] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.3:42585
I20250831 07:58:30.427273 31893 tablet_copy_service.cc:140] P 8247a38b550649aa9cd6e7668dcaec10: Received BeginTabletCopySession request for tablet 59e33bfa463944f89cfbb00d1247323a from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:51246)
I20250831 07:58:30.427868 31893 tablet_copy_service.cc:161] P 8247a38b550649aa9cd6e7668dcaec10: Beginning new tablet copy session on tablet 59e33bfa463944f89cfbb00d1247323a from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:51246: session id = 7ecb257174554a26b2c35ad04b0eabae-59e33bfa463944f89cfbb00d1247323a
I20250831 07:58:30.428290 32702 ts_tablet_manager.cc:927] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:58:30.436252 31893 tablet_copy_source_session.cc:215] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:30.440546 32699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e33bfa463944f89cfbb00d1247323a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:30.443302 32702 tablet_copy_client.cc:323] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.1:45027
I20250831 07:58:30.445295 32701 tablet_copy_client.cc:323] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.3:42585
I20250831 07:58:30.445370 31744 tablet_copy_service.cc:140] P a0d845d01c8c4894895f66e3beae298c: Received BeginTabletCopySession request for tablet bb2667d2ec934433ac10d395c1d29db4 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43230)
I20250831 07:58:30.446182 31744 tablet_copy_service.cc:161] P a0d845d01c8c4894895f66e3beae298c: Beginning new tablet copy session on tablet bb2667d2ec934433ac10d395c1d29db4 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43230: session id = 7ecb257174554a26b2c35ad04b0eabae-bb2667d2ec934433ac10d395c1d29db4
I20250831 07:58:30.476389 31893 tablet_copy_service.cc:140] P 8247a38b550649aa9cd6e7668dcaec10: Received BeginTabletCopySession request for tablet fcbe3794f3fc40819ffbbfef02a071c3 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:51246)
I20250831 07:58:30.476953 31893 tablet_copy_service.cc:161] P 8247a38b550649aa9cd6e7668dcaec10: Beginning new tablet copy session on tablet fcbe3794f3fc40819ffbbfef02a071c3 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:51246: session id = 7ecb257174554a26b2c35ad04b0eabae-fcbe3794f3fc40819ffbbfef02a071c3
I20250831 07:58:30.476498 31744 tablet_copy_source_session.cc:215] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:30.484977 31893 tablet_copy_source_session.cc:215] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:30.504397 32699 tablet_copy_client.cc:806] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:30.505133 32699 tablet_copy_client.cc:670] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:30.510080 32702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb2667d2ec934433ac10d395c1d29db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:30.516637 32699 tablet_copy_client.cc:538] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:30.526857 32701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcbe3794f3fc40819ffbbfef02a071c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:30.633965 32702 tablet_copy_client.cc:806] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:30.634508 32702 tablet_copy_client.cc:670] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:30.641270 32699 tablet_bootstrap.cc:492] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:30.647944 32702 tablet_copy_client.cc:538] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:30.659320 32702 tablet_bootstrap.cc:492] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:30.660863 32701 tablet_copy_client.cc:806] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:30.662002 32701 tablet_copy_client.cc:670] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:30.666990 32705 ts_tablet_manager.cc:927] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:58:30.684813 32705 tablet_copy_client.cc:323] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.1:45027
I20250831 07:58:30.686969 31744 tablet_copy_service.cc:140] P a0d845d01c8c4894895f66e3beae298c: Received BeginTabletCopySession request for tablet 2acd7b3ac5f94f53ac1ef93264415686 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43230)
I20250831 07:58:30.687528 31744 tablet_copy_service.cc:161] P a0d845d01c8c4894895f66e3beae298c: Beginning new tablet copy session on tablet 2acd7b3ac5f94f53ac1ef93264415686 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43230: session id = 7ecb257174554a26b2c35ad04b0eabae-2acd7b3ac5f94f53ac1ef93264415686
I20250831 07:58:30.679394 32701 tablet_copy_client.cc:538] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:30.709558 32701 tablet_bootstrap.cc:492] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:30.722160 32699 tablet_bootstrap.cc:492] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:30.723209 32699 tablet_bootstrap.cc:492] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:30.725340 32699 ts_tablet_manager.cc:1397] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.088s user 0.029s sys 0.012s
I20250831 07:58:30.727283 31744 tablet_copy_source_session.cc:215] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:30.729893 32699 raft_consensus.cc:357] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:30.742221 32699 raft_consensus.cc:738] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:30.742899 32699 consensus_queue.cc:260] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:30.749234 32702 tablet_bootstrap.cc:492] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:30.750245 32702 tablet_bootstrap.cc:492] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:30.751058 32702 ts_tablet_manager.cc:1397] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.092s user 0.024s sys 0.000s
I20250831 07:58:30.753605 32705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2acd7b3ac5f94f53ac1ef93264415686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:30.760615 32699 ts_tablet_manager.cc:1428] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.035s user 0.010s sys 0.004s
I20250831 07:58:30.763895 31893 tablet_copy_service.cc:342] P 8247a38b550649aa9cd6e7668dcaec10: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-59e33bfa463944f89cfbb00d1247323a received from {username='slave'} at 127.0.0.1:51246
I20250831 07:58:30.764654 31893 tablet_copy_service.cc:434] P 8247a38b550649aa9cd6e7668dcaec10: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-59e33bfa463944f89cfbb00d1247323a on tablet 59e33bfa463944f89cfbb00d1247323a with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:30.781980 32467 raft_consensus.cc:1215] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:30.789002 32702 raft_consensus.cc:357] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:30.790681 32702 raft_consensus.cc:738] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:30.792742 32705 tablet_copy_client.cc:806] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:30.793586 32705 tablet_copy_client.cc:670] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:30.793162 32702 consensus_queue.cc:260] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:30.796169 32702 ts_tablet_manager.cc:1428] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.045s user 0.010s sys 0.000s
I20250831 07:58:30.797272 32705 tablet_copy_client.cc:538] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:30.797710 31744 tablet_copy_service.cc:342] P a0d845d01c8c4894895f66e3beae298c: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-bb2667d2ec934433ac10d395c1d29db4 received from {username='slave'} at 127.0.0.1:43230
I20250831 07:58:30.798180 31744 tablet_copy_service.cc:434] P a0d845d01c8c4894895f66e3beae298c: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-bb2667d2ec934433ac10d395c1d29db4 on tablet bb2667d2ec934433ac10d395c1d29db4 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:30.816394 32705 tablet_bootstrap.cc:492] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:30.828943 32701 tablet_bootstrap.cc:492] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:30.829859 32701 tablet_bootstrap.cc:492] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:30.830837 32701 ts_tablet_manager.cc:1397] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.121s user 0.024s sys 0.016s
I20250831 07:58:30.835003 32701 raft_consensus.cc:357] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:30.835983 32701 raft_consensus.cc:738] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:30.836650 32701 consensus_queue.cc:260] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:30.842545 32701 ts_tablet_manager.cc:1428] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.011s user 0.005s sys 0.002s
I20250831 07:58:30.846105 32467 raft_consensus.cc:1215] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:30.848390 31893 tablet_copy_service.cc:342] P 8247a38b550649aa9cd6e7668dcaec10: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-fcbe3794f3fc40819ffbbfef02a071c3 received from {username='slave'} at 127.0.0.1:51246
I20250831 07:58:30.848847 31893 tablet_copy_service.cc:434] P 8247a38b550649aa9cd6e7668dcaec10: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-fcbe3794f3fc40819ffbbfef02a071c3 on tablet fcbe3794f3fc40819ffbbfef02a071c3 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:30.860800 32467 raft_consensus.cc:1215] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:30.868113 32705 tablet_bootstrap.cc:492] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:30.869431 32705 tablet_bootstrap.cc:492] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:30.870162 32705 ts_tablet_manager.cc:1397] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.054s user 0.022s sys 0.001s
I20250831 07:58:30.882756 32705 raft_consensus.cc:357] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:30.883807 32705 raft_consensus.cc:738] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:30.884485 32705 consensus_queue.cc:260] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:30.897222 32705 ts_tablet_manager.cc:1428] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.027s user 0.005s sys 0.001s
I20250831 07:58:30.901314 31744 tablet_copy_service.cc:342] P a0d845d01c8c4894895f66e3beae298c: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-2acd7b3ac5f94f53ac1ef93264415686 received from {username='slave'} at 127.0.0.1:43230
I20250831 07:58:30.901765 31744 tablet_copy_service.cc:434] P a0d845d01c8c4894895f66e3beae298c: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-2acd7b3ac5f94f53ac1ef93264415686 on tablet 2acd7b3ac5f94f53ac1ef93264415686 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:31.012312 32467 raft_consensus.cc:1215] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:31.173832 32654 raft_consensus.cc:1062] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:31.176265 32654 consensus_queue.cc:237] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:31.201858 32467 raft_consensus.cc:1273] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:31.202255 31801 raft_consensus.cc:1273] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:31.203380 32654 consensus_queue.cc:1035] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:31.204406 31733 raft_consensus.cc:1273] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:31.206022 32706 consensus_queue.cc:1035] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:31.209599 32681 consensus_queue.cc:1035] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:31.265736 32672 raft_consensus.cc:1062] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:31.269713 32672 consensus_queue.cc:237] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:31.273443 32706 raft_consensus.cc:2953] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:31.286338 31728 raft_consensus.cc:2953] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:31.292794 32467 raft_consensus.cc:2953] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:31.291778 31801 raft_consensus.cc:2953] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:31.309727 31630 catalog_manager.cc:5582] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:31.319519 32470 raft_consensus.cc:1273] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:31.322933 32662 consensus_queue.cc:1035] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:31.331311 32654 raft_consensus.cc:1062] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:31.334283 32654 consensus_queue.cc:237] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:31.343588 31877 raft_consensus.cc:1273] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:31.345959 32662 consensus_queue.cc:1035] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:31.356243 31802 raft_consensus.cc:1273] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:31.360659 32671 raft_consensus.cc:2953] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:31.368677 31877 raft_consensus.cc:2953] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:31.378996 32607 consensus_queue.cc:1035] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250831 07:58:31.484361 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 129208us
I20250831 07:58:31.485443 31802 raft_consensus.cc:2953] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:31.609798 31801 raft_consensus.cc:1273] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:31.618379 31730 raft_consensus.cc:1273] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:31.627005 32467 raft_consensus.cc:1273] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:31.636839 32715 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:31.652585 32468 raft_consensus.cc:2953] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:31.689951 32736 raft_consensus.cc:1062] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:31.700129 32736 consensus_queue.cc:237] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:31.703894 31629 catalog_manager.cc:5582] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:31.723639 32743 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250831 07:58:31.725507 32468 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:31.726922 32737 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250831 07:58:31.737154 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 114672us
W20250831 07:58:31.830531 32747 log.cc:927] Time spent T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20250831 07:58:31.909523 31882 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250831 07:58:31.911850 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 149786us
W20250831 07:58:31.947975 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 149060us
I20250831 07:58:31.972497 31809 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:31.977667 31877 consensus_queue.cc:237] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:31.995018 32744 raft_consensus.cc:2953] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:31.997726 31804 raft_consensus.cc:1273] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:31.997406 31802 raft_consensus.cc:2953] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:31.998972 32723 consensus_queue.cc:1035] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250831 07:58:32.004937 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:32.007491 32742 raft_consensus.cc:2953] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.011729 32758 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:32.008210 31800 raft_consensus.cc:2953] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.019734 32724 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:32.037935 31616 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 59e33bfa463944f89cfbb00d1247323a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:32.048462 31628 catalog_manager.cc:5582] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:32.091298 31630 catalog_manager.cc:5582] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.100855 31734 consensus_queue.cc:237] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:32.103842 31716 tablet_service.cc:1515] Processing DeleteTablet for tablet 59e33bfa463944f89cfbb00d1247323a with delete_type TABLET_DATA_TOMBSTONED (TS a0d845d01c8c4894895f66e3beae298c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57666
I20250831 07:58:32.107630 31728 raft_consensus.cc:2953] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.117875 32470 raft_consensus.cc:1273] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:32.137842 32662 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:32.190892 32706 consensus_queue.cc:1035] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20250831 07:58:32.231285 32470 raft_consensus.cc:2953] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.291330 32468 raft_consensus.cc:2953] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
W20250831 07:58:32.296722 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 104461us
W20250831 07:58:32.318341 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 128437us
I20250831 07:58:32.319208 32764 tablet_replica.cc:331] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c: stopping tablet replica
I20250831 07:58:32.341991 32764 raft_consensus.cc:2241] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:32.343057 32764 raft_consensus.cc:2270] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:32.352594 31882 raft_consensus.cc:1273] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:32.354261 32470 raft_consensus.cc:1273] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:32.369660 32764 ts_tablet_manager.cc:1905] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:32.394831 31800 raft_consensus.cc:2953] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.389812 32767 raft_consensus.cc:2953] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.408046 32736 consensus_queue.cc:1035] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:58:32.431675 31630 catalog_manager.cc:5582] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.443951 32738 consensus_queue.cc:1035] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20250831 07:58:32.453428 31882 raft_consensus.cc:2953] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.461839 31728 consensus_queue.cc:237] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:32.461318 32468 raft_consensus.cc:2953] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.471568 32672 raft_consensus.cc:2953] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.473412 32470 raft_consensus.cc:2953] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.493753 31618 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet bb2667d2ec934433ac10d395c1d29db4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:32.507656 32764 ts_tablet_manager.cc:1918] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:32.514070 31800 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:32.517678 32764 log.cc:1199] T 59e33bfa463944f89cfbb00d1247323a P a0d845d01c8c4894895f66e3beae298c: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-0-root/wals/59e33bfa463944f89cfbb00d1247323a
I20250831 07:58:32.520817 32468 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:32.523236 31618 catalog_manager.cc:4928] TS a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): tablet 59e33bfa463944f89cfbb00d1247323a (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:32.535713 32729 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:58:32.545837 31883 raft_consensus.cc:2953] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.546074 31630 catalog_manager.cc:5582] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.548972 32738 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:58:32.550495 32672 raft_consensus.cc:2953] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.566124 31809 raft_consensus.cc:2953] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.582139 31618 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 2acd7b3ac5f94f53ac1ef93264415686 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250831 07:58:32.589143 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 44027.3us
I20250831 07:58:32.629055 32470 raft_consensus.cc:2953] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
W20250831 07:58:32.635409 31618 catalog_manager.cc:5105] ChangeConfig:REMOVE_PEER RPC for tablet 2acd7b3ac5f94f53ac1ef93264415686 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20250831 07:58:32.638065 31791 tablet_service.cc:1515] Processing DeleteTablet for tablet bb2667d2ec934433ac10d395c1d29db4 with delete_type TABLET_DATA_TOMBSTONED (TS 7512ba2ed3a94982a22a75f51a647e65 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39068
I20250831 07:58:32.645210 31630 catalog_manager.cc:5582] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.678494 31873 tablet_service.cc:1515] Processing DeleteTablet for tablet 2acd7b3ac5f94f53ac1ef93264415686 with delete_type TABLET_DATA_TOMBSTONED (TS 8247a38b550649aa9cd6e7668dcaec10 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57332
I20250831 07:58:32.679013 304 tablet_replica.cc:331] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65: stopping tablet replica
I20250831 07:58:32.708087 31882 consensus_queue.cc:237] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:32.712246 304 raft_consensus.cc:2241] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:32.719534 304 raft_consensus.cc:2270] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:32.740172 304 ts_tablet_manager.cc:1905] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:32.748239 32470 raft_consensus.cc:1273] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:32.750440 32706 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:32.753342 305 tablet_replica.cc:331] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10: stopping tablet replica
I20250831 07:58:32.756297 305 raft_consensus.cc:2241] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:32.762522 305 raft_consensus.cc:2270] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:32.755275 31800 raft_consensus.cc:1273] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:32.768249 32742 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250831 07:58:32.772773 32755 raft_consensus.cc:2953] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.781894 32470 raft_consensus.cc:2953] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.789557 305 ts_tablet_manager.cc:1905] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:32.820137 31616 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet fcbe3794f3fc40819ffbbfef02a071c3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:32.825279 304 ts_tablet_manager.cc:1918] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:32.825795 304 log.cc:1199] T bb2667d2ec934433ac10d395c1d29db4 P 7512ba2ed3a94982a22a75f51a647e65: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-1-root/wals/bb2667d2ec934433ac10d395c1d29db4
I20250831 07:58:32.828375 31631 catalog_manager.cc:5582] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.839622 31617 catalog_manager.cc:4928] TS 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): tablet bb2667d2ec934433ac10d395c1d29db4 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:32.841503 31800 raft_consensus.cc:2953] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:32.859992 31716 tablet_service.cc:1515] Processing DeleteTablet for tablet fcbe3794f3fc40819ffbbfef02a071c3 with delete_type TABLET_DATA_TOMBSTONED (TS a0d845d01c8c4894895f66e3beae298c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57666
I20250831 07:58:32.861088 32764 tablet_replica.cc:331] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c: stopping tablet replica
I20250831 07:58:32.862001 32764 raft_consensus.cc:2241] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:32.862715 32764 raft_consensus.cc:2270] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:32.870963 305 ts_tablet_manager.cc:1918] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:32.871546 305 log.cc:1199] T 2acd7b3ac5f94f53ac1ef93264415686 P 8247a38b550649aa9cd6e7668dcaec10: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-2-root/wals/2acd7b3ac5f94f53ac1ef93264415686
I20250831 07:58:32.869736 32764 ts_tablet_manager.cc:1905] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:32.873361 31616 catalog_manager.cc:4928] TS 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): tablet 2acd7b3ac5f94f53ac1ef93264415686 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:32.895016 32764 ts_tablet_manager.cc:1918] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:32.895689 32764 log.cc:1199] T fcbe3794f3fc40819ffbbfef02a071c3 P a0d845d01c8c4894895f66e3beae298c: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-0-root/wals/fcbe3794f3fc40819ffbbfef02a071c3
I20250831 07:58:32.897375 31618 catalog_manager.cc:4928] TS a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): tablet fcbe3794f3fc40819ffbbfef02a071c3 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:33.946121 31728 consensus_queue.cc:237] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
W20250831 07:58:33.957326 31698 consensus_peers.cc:489] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 000f9ab653554f498fe14065e6656a62. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:33.958539 31882 raft_consensus.cc:1273] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:33.960558 310 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:33.964385 31802 raft_consensus.cc:1273] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:33.967486 310 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:33.979327 32767 raft_consensus.cc:2953] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:33.986589 31800 raft_consensus.cc:2953] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:33.996172 31883 raft_consensus.cc:2953] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:34.005822 31882 consensus_queue.cc:237] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:34.016408 31631 catalog_manager.cc:5582] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:34.044606 31800 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:34.046142 32742 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:58:34.049193 31847 consensus_peers.cc:489] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a40df3e72bd462091ca307d230a90ed. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:34.050254 31728 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:34.051512 306 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:34.091076 31800 raft_consensus.cc:2953] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:34.088121 32742 raft_consensus.cc:2953] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:34.126866 31728 raft_consensus.cc:2953] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:34.138338 31630 catalog_manager.cc:5582] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:34.164422 31800 consensus_queue.cc:237] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
W20250831 07:58:34.234200 31773 consensus_peers.cc:489] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 518e6f5a91554de78e0bd0b6ec04c12a. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:34.234480 31882 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:34.235347 31728 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:34.237068 307 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:34.240042 307 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:34.265430 308 raft_consensus.cc:2953] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:34.268872 31882 raft_consensus.cc:2953] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:34.278131 31728 raft_consensus.cc:2953] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:34.288569 31630 catalog_manager.cc:5582] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:34.333485 31882 consensus_queue.cc:237] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
W20250831 07:58:34.369436 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 73450.4us
I20250831 07:58:34.376237 31800 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:34.377727 32755 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:34.383620 31728 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250831 07:58:34.384346 31847 consensus_peers.cc:489] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe2bab0189fe475cb540bdf7a041d8b8. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:34.385062 32755 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:34.401782 32755 raft_consensus.cc:2953] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:34.409133 31802 raft_consensus.cc:2953] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:34.426301 31630 catalog_manager.cc:5582] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:34.453329 31728 raft_consensus.cc:2953] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:34.526434 329 ts_tablet_manager.cc:927] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:58:34.538110 329 tablet_copy_client.cc:323] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.1:45027
I20250831 07:58:34.540664 31744 tablet_copy_service.cc:140] P a0d845d01c8c4894895f66e3beae298c: Received BeginTabletCopySession request for tablet 000f9ab653554f498fe14065e6656a62 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43230)
I20250831 07:58:34.541224 31744 tablet_copy_service.cc:161] P a0d845d01c8c4894895f66e3beae298c: Beginning new tablet copy session on tablet 000f9ab653554f498fe14065e6656a62 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43230: session id = 7ecb257174554a26b2c35ad04b0eabae-000f9ab653554f498fe14065e6656a62
I20250831 07:58:34.549338 31744 tablet_copy_source_session.cc:215] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:34.552546 329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 000f9ab653554f498fe14065e6656a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:34.610097 329 tablet_copy_client.cc:806] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:34.611032 329 tablet_copy_client.cc:670] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:34.615034 329 tablet_copy_client.cc:538] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:34.625564 329 tablet_bootstrap.cc:492] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:34.630323 331 ts_tablet_manager.cc:927] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:34.642122 331 tablet_copy_client.cc:323] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.3:42585
I20250831 07:58:34.648404 31893 tablet_copy_service.cc:140] P 8247a38b550649aa9cd6e7668dcaec10: Received BeginTabletCopySession request for tablet 9a40df3e72bd462091ca307d230a90ed from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:51246)
I20250831 07:58:34.649154 31893 tablet_copy_service.cc:161] P 8247a38b550649aa9cd6e7668dcaec10: Beginning new tablet copy session on tablet 9a40df3e72bd462091ca307d230a90ed from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:51246: session id = 7ecb257174554a26b2c35ad04b0eabae-9a40df3e72bd462091ca307d230a90ed
I20250831 07:58:34.666025 333 ts_tablet_manager.cc:927] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:34.834865 31893 tablet_copy_source_session.cc:215] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:34.900816 336 ts_tablet_manager.cc:927] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:34.913244 336 tablet_copy_client.cc:323] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.3:42585
I20250831 07:58:34.921527 333 tablet_copy_client.cc:323] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.2:37261
I20250831 07:58:34.923461 31819 tablet_copy_service.cc:140] P 7512ba2ed3a94982a22a75f51a647e65: Received BeginTabletCopySession request for tablet 518e6f5a91554de78e0bd0b6ec04c12a from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43428)
I20250831 07:58:34.924001 31819 tablet_copy_service.cc:161] P 7512ba2ed3a94982a22a75f51a647e65: Beginning new tablet copy session on tablet 518e6f5a91554de78e0bd0b6ec04c12a from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43428: session id = 7ecb257174554a26b2c35ad04b0eabae-518e6f5a91554de78e0bd0b6ec04c12a
I20250831 07:58:34.932339 31819 tablet_copy_source_session.cc:215] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:34.936051 333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 518e6f5a91554de78e0bd0b6ec04c12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:34.983920 333 tablet_copy_client.cc:806] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:34.984658 333 tablet_copy_client.cc:670] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:35.002032 333 tablet_copy_client.cc:538] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:35.025352 331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a40df3e72bd462091ca307d230a90ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:35.044559 31893 tablet_copy_service.cc:140] P 8247a38b550649aa9cd6e7668dcaec10: Received BeginTabletCopySession request for tablet fe2bab0189fe475cb540bdf7a041d8b8 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:51246)
I20250831 07:58:35.045362 31893 tablet_copy_service.cc:161] P 8247a38b550649aa9cd6e7668dcaec10: Beginning new tablet copy session on tablet fe2bab0189fe475cb540bdf7a041d8b8 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:51246: session id = 7ecb257174554a26b2c35ad04b0eabae-fe2bab0189fe475cb540bdf7a041d8b8
I20250831 07:58:35.070720 329 tablet_bootstrap.cc:492] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:35.071848 329 tablet_bootstrap.cc:492] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:35.072609 329 ts_tablet_manager.cc:1397] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.447s user 0.118s sys 0.252s
I20250831 07:58:35.078966 333 tablet_bootstrap.cc:492] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:35.075587 329 raft_consensus.cc:357] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:35.084918 329 raft_consensus.cc:738] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:35.085677 329 consensus_queue.cc:260] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:35.090565 31893 tablet_copy_source_session.cc:215] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:35.100674 331 tablet_copy_client.cc:806] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:35.110541 329 ts_tablet_manager.cc:1428] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.038s user 0.004s sys 0.012s
I20250831 07:58:35.113350 31744 tablet_copy_service.cc:342] P a0d845d01c8c4894895f66e3beae298c: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-000f9ab653554f498fe14065e6656a62 received from {username='slave'} at 127.0.0.1:43230
I20250831 07:58:35.113771 31744 tablet_copy_service.cc:434] P a0d845d01c8c4894895f66e3beae298c: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-000f9ab653554f498fe14065e6656a62 on tablet 000f9ab653554f498fe14065e6656a62 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:35.110936 331 tablet_copy_client.cc:670] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:35.146970 336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe2bab0189fe475cb540bdf7a041d8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:35.160437 331 tablet_copy_client.cc:538] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:35.177273 331 tablet_bootstrap.cc:492] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:35.179217 333 tablet_bootstrap.cc:492] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:35.187883 333 tablet_bootstrap.cc:492] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:35.188728 333 ts_tablet_manager.cc:1397] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.142s user 0.024s sys 0.008s
I20250831 07:58:35.192318 333 raft_consensus.cc:357] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:35.193292 333 raft_consensus.cc:738] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:35.194217 333 consensus_queue.cc:260] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:35.197284 333 ts_tablet_manager.cc:1428] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250831 07:58:35.199127 31819 tablet_copy_service.cc:342] P 7512ba2ed3a94982a22a75f51a647e65: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-518e6f5a91554de78e0bd0b6ec04c12a received from {username='slave'} at 127.0.0.1:43428
I20250831 07:58:35.199568 31819 tablet_copy_service.cc:434] P 7512ba2ed3a94982a22a75f51a647e65: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-518e6f5a91554de78e0bd0b6ec04c12a on tablet 518e6f5a91554de78e0bd0b6ec04c12a with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:35.222918 336 tablet_copy_client.cc:806] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:35.223981 336 tablet_copy_client.cc:670] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:35.246551 336 tablet_copy_client.cc:538] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:35.259799 331 tablet_bootstrap.cc:492] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:35.261031 331 tablet_bootstrap.cc:492] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:35.261811 331 ts_tablet_manager.cc:1397] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.085s user 0.024s sys 0.024s
I20250831 07:58:35.264947 331 raft_consensus.cc:357] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:35.265940 331 raft_consensus.cc:738] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:35.266685 331 consensus_queue.cc:260] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:35.269289 331 ts_tablet_manager.cc:1428] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 07:58:35.272920 336 tablet_bootstrap.cc:492] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:35.287037 31893 tablet_copy_service.cc:342] P 8247a38b550649aa9cd6e7668dcaec10: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-9a40df3e72bd462091ca307d230a90ed received from {username='slave'} at 127.0.0.1:51246
I20250831 07:58:35.287607 31893 tablet_copy_service.cc:434] P 8247a38b550649aa9cd6e7668dcaec10: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-9a40df3e72bd462091ca307d230a90ed on tablet 9a40df3e72bd462091ca307d230a90ed with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:35.313361 336 tablet_bootstrap.cc:492] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:35.314952 336 tablet_bootstrap.cc:492] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:35.315681 336 ts_tablet_manager.cc:1397] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.043s user 0.022s sys 0.010s
I20250831 07:58:35.320600 336 raft_consensus.cc:357] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:35.321492 336 raft_consensus.cc:738] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:35.324131 336 consensus_queue.cc:260] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:35.327225 336 ts_tablet_manager.cc:1428] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250831 07:58:35.332617 31893 tablet_copy_service.cc:342] P 8247a38b550649aa9cd6e7668dcaec10: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-fe2bab0189fe475cb540bdf7a041d8b8 received from {username='slave'} at 127.0.0.1:51246
I20250831 07:58:35.333084 31893 tablet_copy_service.cc:434] P 8247a38b550649aa9cd6e7668dcaec10: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-fe2bab0189fe475cb540bdf7a041d8b8 on tablet fe2bab0189fe475cb540bdf7a041d8b8 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:35.494623 32467 raft_consensus.cc:1215] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:35.542379 32467 raft_consensus.cc:1215] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:35.567444 32467 raft_consensus.cc:1215] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:35.685125 32467 raft_consensus.cc:1215] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:35.953271 32742 raft_consensus.cc:1062] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:35.955511 32742 consensus_queue.cc:237] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:35.967278 32467 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:35.968875 32742 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:35.968972 31727 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:35.971526 32742 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:35.975426 31802 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:35.978598 32742 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:36.013758 306 raft_consensus.cc:2953] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.023196 32742 raft_consensus.cc:1062] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:36.025442 32742 consensus_queue.cc:237] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:36.032625 32467 raft_consensus.cc:2953] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.042307 31805 raft_consensus.cc:2953] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.060487 31809 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:36.062539 31728 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:36.062657 31727 raft_consensus.cc:2953] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.065258 32470 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:36.067224 31628 catalog_manager.cc:5582] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:36.079453 32742 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:36.082825 353 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:36.084398 306 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:36.151860 306 raft_consensus.cc:2953] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.164955 32468 raft_consensus.cc:2953] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.167680 32767 raft_consensus.cc:1062] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:36.170239 32767 consensus_queue.cc:237] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:36.188968 31877 raft_consensus.cc:1273] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:36.190266 310 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:36.192366 31728 raft_consensus.cc:2953] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.194235 32468 raft_consensus.cc:1273] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:36.197930 32767 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:36.200074 31877 consensus_queue.cc:237] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:36.205687 31802 raft_consensus.cc:1273] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:36.211943 32767 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:36.223372 347 raft_consensus.cc:1062] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:36.226764 31629 catalog_manager.cc:5582] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.226195 347 consensus_queue.cc:237] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:36.242523 32767 raft_consensus.cc:2953] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.244134 31883 raft_consensus.cc:2953] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.249924 32470 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:36.250286 31725 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:36.251696 31881 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:36.253525 32470 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:36.253955 347 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:36.255246 346 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:36.256489 32755 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:36.256927 31725 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:36.258040 353 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:36.259660 347 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:36.265892 32755 raft_consensus.cc:2953] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.270635 32468 raft_consensus.cc:2953] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.274631 31730 raft_consensus.cc:2953] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.277269 32467 raft_consensus.cc:2953] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.311517 31631 catalog_manager.cc:5582] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:36.312170 31809 raft_consensus.cc:2953] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.338583 31616 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet fe2bab0189fe475cb540bdf7a041d8b8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:36.346797 348 raft_consensus.cc:2953] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.352283 31727 raft_consensus.cc:2953] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.362793 31802 raft_consensus.cc:2953] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.376683 32468 raft_consensus.cc:2953] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.377179 31881 raft_consensus.cc:2953] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.396934 31629 catalog_manager.cc:5582] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.412945 31631 catalog_manager.cc:5582] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:36.421478 31791 tablet_service.cc:1515] Processing DeleteTablet for tablet fe2bab0189fe475cb540bdf7a041d8b8 with delete_type TABLET_DATA_TOMBSTONED (TS 7512ba2ed3a94982a22a75f51a647e65 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39068
I20250831 07:58:36.417834 31881 consensus_queue.cc:237] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:36.426445 372 tablet_replica.cc:331] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65: stopping tablet replica
I20250831 07:58:36.427470 372 raft_consensus.cc:2241] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:36.428275 372 raft_consensus.cc:2270] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:36.434592 32468 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:36.434749 31730 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:36.436096 32742 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:36.437381 372 ts_tablet_manager.cc:1905] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:36.437197 355 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:36.439054 31809 consensus_queue.cc:237] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:36.446525 356 raft_consensus.cc:2953] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.449982 31730 raft_consensus.cc:2953] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.455478 31727 consensus_queue.cc:237] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:36.458014 32468 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:36.457593 32467 raft_consensus.cc:2953] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.459183 348 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:36.460076 31616 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9a40df3e72bd462091ca307d230a90ed with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:36.470396 31627 catalog_manager.cc:5582] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:36.472654 32471 raft_consensus.cc:1273] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:36.474553 310 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:36.475230 31881 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:36.476259 307 raft_consensus.cc:2953] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.480993 347 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:36.488993 31881 raft_consensus.cc:2953] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.493100 310 raft_consensus.cc:2953] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.494315 32471 raft_consensus.cc:2953] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.485452 372 ts_tablet_manager.cc:1918] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:36.498317 31617 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 518e6f5a91554de78e0bd0b6ec04c12a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:36.488803 32468 raft_consensus.cc:2953] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.509919 31802 raft_consensus.cc:1273] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:36.511874 372 log.cc:1199] T fe2bab0189fe475cb540bdf7a041d8b8 P 7512ba2ed3a94982a22a75f51a647e65: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-1-root/wals/fe2bab0189fe475cb540bdf7a041d8b8
I20250831 07:58:36.515069 31791 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a40df3e72bd462091ca307d230a90ed with delete_type TABLET_DATA_TOMBSTONED (TS 7512ba2ed3a94982a22a75f51a647e65 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39068
I20250831 07:58:36.517572 31618 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 000f9ab653554f498fe14065e6656a62 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:36.519953 31617 catalog_manager.cc:4928] TS 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): tablet fe2bab0189fe475cb540bdf7a041d8b8 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:36.520654 372 tablet_replica.cc:331] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65: stopping tablet replica
I20250831 07:58:36.521731 372 raft_consensus.cc:2241] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:36.521304 309 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250831 07:58:36.521185 31630 catalog_manager.cc:5582] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:36.522608 372 raft_consensus.cc:2270] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:36.527355 372 ts_tablet_manager.cc:1905] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:36.552296 31630 catalog_manager.cc:5582] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.554137 31716 tablet_service.cc:1515] Processing DeleteTablet for tablet 518e6f5a91554de78e0bd0b6ec04c12a with delete_type TABLET_DATA_TOMBSTONED (TS a0d845d01c8c4894895f66e3beae298c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57666
I20250831 07:58:36.540587 31809 raft_consensus.cc:2953] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:36.562414 374 tablet_replica.cc:331] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c: stopping tablet replica
I20250831 07:58:36.567456 374 raft_consensus.cc:2241] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:36.568313 374 raft_consensus.cc:2270] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:36.577678 31873 tablet_service.cc:1515] Processing DeleteTablet for tablet 000f9ab653554f498fe14065e6656a62 with delete_type TABLET_DATA_TOMBSTONED (TS 8247a38b550649aa9cd6e7668dcaec10 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57332
I20250831 07:58:36.589728 374 ts_tablet_manager.cc:1905] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:36.593942 375 tablet_replica.cc:331] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10: stopping tablet replica
I20250831 07:58:36.596942 375 raft_consensus.cc:2241] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:36.597720 375 raft_consensus.cc:2270] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:36.600713 372 ts_tablet_manager.cc:1918] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:36.600894 375 ts_tablet_manager.cc:1905] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:36.601325 372 log.cc:1199] T 9a40df3e72bd462091ca307d230a90ed P 7512ba2ed3a94982a22a75f51a647e65: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-1-root/wals/9a40df3e72bd462091ca307d230a90ed
I20250831 07:58:36.602815 31617 catalog_manager.cc:4928] TS 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): tablet 9a40df3e72bd462091ca307d230a90ed (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:36.608812 374 ts_tablet_manager.cc:1918] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:36.609360 374 log.cc:1199] T 518e6f5a91554de78e0bd0b6ec04c12a P a0d845d01c8c4894895f66e3beae298c: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-0-root/wals/518e6f5a91554de78e0bd0b6ec04c12a
I20250831 07:58:36.612200 31618 catalog_manager.cc:4928] TS a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): tablet 518e6f5a91554de78e0bd0b6ec04c12a (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:36.615854 375 ts_tablet_manager.cc:1918] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:36.616356 375 log.cc:1199] T 000f9ab653554f498fe14065e6656a62 P 8247a38b550649aa9cd6e7668dcaec10: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-2-root/wals/000f9ab653554f498fe14065e6656a62
I20250831 07:58:36.618547 31616 catalog_manager.cc:4928] TS 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): tablet 000f9ab653554f498fe14065e6656a62 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
W20250831 07:58:37.320079 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 381435us
W20250831 07:58:37.427621 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 128802us
W20250831 07:58:37.541507 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 112139us
I20250831 07:58:37.718792 31802 consensus_queue.cc:237] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:37.726039 31878 raft_consensus.cc:1273] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:37.726579 31727 raft_consensus.cc:1273] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250831 07:58:37.727413 31773 consensus_peers.cc:489] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5eac48bce84e483582f863716d5c26f9. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:37.727946 348 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:37.728847 376 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:37.741415 376 raft_consensus.cc:2953] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:37.743130 31727 raft_consensus.cc:2953] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:37.755165 31883 raft_consensus.cc:2953] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:37.779369 31627 catalog_manager.cc:5582] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:37.783576 31881 consensus_queue.cc:237] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:37.823634 31802 raft_consensus.cc:1273] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:37.823917 31728 raft_consensus.cc:1273] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:37.825812 397 consensus_queue.cc:1035] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250831 07:58:37.831081 31847 consensus_peers.cc:489] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cbf2ae2189542948a3ffb6b88a4fa21. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:37.833834 397 consensus_queue.cc:1035] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:37.866165 387 raft_consensus.cc:2953] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:37.875070 31728 raft_consensus.cc:2953] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:37.905496 31804 raft_consensus.cc:2953] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:37.926436 31881 consensus_queue.cc:237] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:37.949430 31628 catalog_manager.cc:5582] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:37.953688 31805 raft_consensus.cc:1273] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:37.955636 31727 raft_consensus.cc:1273] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250831 07:58:37.957301 31847 consensus_peers.cc:489] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a187e6c5ed8f4214b4486aab25955909. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:37.959568 32755 consensus_queue.cc:1035] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250831 07:58:37.967326 389 consensus_queue.cc:1035] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:37.986867 389 raft_consensus.cc:2953] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:37.989537 31805 raft_consensus.cc:2953] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
W20250831 07:58:37.993139 31830 debug-util.cc:398] Leaking SignalData structure 0x7b08001b87c0 after lost signal to thread 30567
W20250831 07:58:37.994426 31830 debug-util.cc:398] Leaking SignalData structure 0x7b08000ec380 after lost signal to thread 31666
W20250831 07:58:37.995898 31830 debug-util.cc:398] Leaking SignalData structure 0x7b0800344220 after lost signal to thread 31758
W20250831 07:58:37.997131 31830 debug-util.cc:398] Leaking SignalData structure 0x7b08001ca960 after lost signal to thread 31833
I20250831 07:58:37.996279 31728 raft_consensus.cc:2953] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
W20250831 07:58:37.998409 31830 debug-util.cc:398] Leaking SignalData structure 0x7b08001dc700 after lost signal to thread 31907
W20250831 07:58:37.999769 31830 debug-util.cc:398] Leaking SignalData structure 0x7b08002df2c0 after lost signal to thread 32495
I20250831 07:58:38.013669 31628 catalog_manager.cc:5582] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:38.021481 31805 consensus_queue.cc:237] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:38.037736 31883 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250831 07:58:38.038425 31773 consensus_peers.cc:489] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90effb264f51485dac220ee32f49e4ad. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:38.040402 348 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:38.045838 31727 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:38.050640 376 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:38.076934 348 raft_consensus.cc:2953] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:38.080453 31725 raft_consensus.cc:2953] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:38.093977 31881 raft_consensus.cc:2953] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:38.107677 31630 catalog_manager.cc:5582] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:38.283190 414 ts_tablet_manager.cc:927] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:38.292559 414 tablet_copy_client.cc:323] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.2:37261
I20250831 07:58:38.294586 31819 tablet_copy_service.cc:140] P 7512ba2ed3a94982a22a75f51a647e65: Received BeginTabletCopySession request for tablet 5eac48bce84e483582f863716d5c26f9 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43428)
I20250831 07:58:38.295230 31819 tablet_copy_service.cc:161] P 7512ba2ed3a94982a22a75f51a647e65: Beginning new tablet copy session on tablet 5eac48bce84e483582f863716d5c26f9 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43428: session id = 7ecb257174554a26b2c35ad04b0eabae-5eac48bce84e483582f863716d5c26f9
I20250831 07:58:38.311758 31819 tablet_copy_source_session.cc:215] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:38.315101 414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eac48bce84e483582f863716d5c26f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:38.334677 414 tablet_copy_client.cc:806] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:38.337155 414 tablet_copy_client.cc:670] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:38.343302 414 tablet_copy_client.cc:538] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:38.362989 414 tablet_bootstrap.cc:492] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:38.392294 414 tablet_bootstrap.cc:492] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:38.393067 414 tablet_bootstrap.cc:492] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:38.393638 414 ts_tablet_manager.cc:1397] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.014s
I20250831 07:58:38.396595 414 raft_consensus.cc:357] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:38.397421 414 raft_consensus.cc:738] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:38.397965 414 consensus_queue.cc:260] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:38.401412 414 ts_tablet_manager.cc:1428] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20250831 07:58:38.406668 31819 tablet_copy_service.cc:342] P 7512ba2ed3a94982a22a75f51a647e65: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-5eac48bce84e483582f863716d5c26f9 received from {username='slave'} at 127.0.0.1:43428
I20250831 07:58:38.407155 31819 tablet_copy_service.cc:434] P 7512ba2ed3a94982a22a75f51a647e65: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-5eac48bce84e483582f863716d5c26f9 on tablet 5eac48bce84e483582f863716d5c26f9 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:38.436427 414 ts_tablet_manager.cc:927] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:38.479914 418 ts_tablet_manager.cc:927] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:38.487880 414 tablet_copy_client.cc:323] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.2:37261
I20250831 07:58:38.490410 31819 tablet_copy_service.cc:140] P 7512ba2ed3a94982a22a75f51a647e65: Received BeginTabletCopySession request for tablet 90effb264f51485dac220ee32f49e4ad from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43428)
I20250831 07:58:38.490936 31819 tablet_copy_service.cc:161] P 7512ba2ed3a94982a22a75f51a647e65: Beginning new tablet copy session on tablet 90effb264f51485dac220ee32f49e4ad from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43428: session id = 7ecb257174554a26b2c35ad04b0eabae-90effb264f51485dac220ee32f49e4ad
I20250831 07:58:38.508587 418 tablet_copy_client.cc:323] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.3:42585
I20250831 07:58:38.513705 31819 tablet_copy_source_session.cc:215] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:38.516887 419 ts_tablet_manager.cc:927] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:38.517086 414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90effb264f51485dac220ee32f49e4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:38.535367 414 tablet_copy_client.cc:806] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:38.536248 414 tablet_copy_client.cc:670] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:38.543641 31893 tablet_copy_service.cc:140] P 8247a38b550649aa9cd6e7668dcaec10: Received BeginTabletCopySession request for tablet 0cbf2ae2189542948a3ffb6b88a4fa21 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:51246)
I20250831 07:58:38.544258 31893 tablet_copy_service.cc:161] P 8247a38b550649aa9cd6e7668dcaec10: Beginning new tablet copy session on tablet 0cbf2ae2189542948a3ffb6b88a4fa21 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:51246: session id = 7ecb257174554a26b2c35ad04b0eabae-0cbf2ae2189542948a3ffb6b88a4fa21
I20250831 07:58:38.552187 414 tablet_copy_client.cc:538] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:38.557081 419 tablet_copy_client.cc:323] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.3:42585
I20250831 07:58:38.560951 31892 tablet_copy_service.cc:140] P 8247a38b550649aa9cd6e7668dcaec10: Received BeginTabletCopySession request for tablet a187e6c5ed8f4214b4486aab25955909 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:51246)
I20250831 07:58:38.561538 31892 tablet_copy_service.cc:161] P 8247a38b550649aa9cd6e7668dcaec10: Beginning new tablet copy session on tablet a187e6c5ed8f4214b4486aab25955909 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:51246: session id = 7ecb257174554a26b2c35ad04b0eabae-a187e6c5ed8f4214b4486aab25955909
I20250831 07:58:38.575891 31892 tablet_copy_source_session.cc:215] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:38.581549 414 tablet_bootstrap.cc:492] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:38.582449 419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a187e6c5ed8f4214b4486aab25955909. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:38.612735 31893 tablet_copy_source_session.cc:215] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:38.619424 418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cbf2ae2189542948a3ffb6b88a4fa21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:38.624934 419 tablet_copy_client.cc:806] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:38.634268 419 tablet_copy_client.cc:670] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:38.639876 419 tablet_copy_client.cc:538] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:38.658347 414 tablet_bootstrap.cc:492] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:38.658525 419 tablet_bootstrap.cc:492] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:38.659802 414 tablet_bootstrap.cc:492] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:38.660831 414 ts_tablet_manager.cc:1397] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.080s user 0.029s sys 0.010s
I20250831 07:58:38.664140 414 raft_consensus.cc:357] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:38.665210 414 raft_consensus.cc:738] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:38.665978 414 consensus_queue.cc:260] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:38.669943 418 tablet_copy_client.cc:806] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:38.670635 418 tablet_copy_client.cc:670] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:38.676599 418 tablet_copy_client.cc:538] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:38.683009 414 ts_tablet_manager.cc:1428] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
I20250831 07:58:38.686260 418 tablet_bootstrap.cc:492] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:38.699981 31819 tablet_copy_service.cc:342] P 7512ba2ed3a94982a22a75f51a647e65: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-90effb264f51485dac220ee32f49e4ad received from {username='slave'} at 127.0.0.1:43428
I20250831 07:58:38.700784 31819 tablet_copy_service.cc:434] P 7512ba2ed3a94982a22a75f51a647e65: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-90effb264f51485dac220ee32f49e4ad on tablet 90effb264f51485dac220ee32f49e4ad with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:38.731426 419 tablet_bootstrap.cc:492] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:38.732434 419 tablet_bootstrap.cc:492] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:38.733132 419 ts_tablet_manager.cc:1397] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.075s user 0.041s sys 0.012s
I20250831 07:58:38.736054 419 raft_consensus.cc:357] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:38.736935 419 raft_consensus.cc:738] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:38.737634 419 consensus_queue.cc:260] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:38.742795 419 ts_tablet_manager.cc:1428] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250831 07:58:38.747567 31893 tablet_copy_service.cc:342] P 8247a38b550649aa9cd6e7668dcaec10: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-a187e6c5ed8f4214b4486aab25955909 received from {username='slave'} at 127.0.0.1:51246
I20250831 07:58:38.748051 31893 tablet_copy_service.cc:434] P 8247a38b550649aa9cd6e7668dcaec10: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-a187e6c5ed8f4214b4486aab25955909 on tablet a187e6c5ed8f4214b4486aab25955909 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:38.759672 418 tablet_bootstrap.cc:492] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:38.760854 418 tablet_bootstrap.cc:492] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:38.761655 418 ts_tablet_manager.cc:1397] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.076s user 0.019s sys 0.009s
I20250831 07:58:38.764619 418 raft_consensus.cc:357] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:38.765508 418 raft_consensus.cc:738] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:38.766218 418 consensus_queue.cc:260] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:38.771859 418 ts_tablet_manager.cc:1428] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20250831 07:58:38.773753 31893 tablet_copy_service.cc:342] P 8247a38b550649aa9cd6e7668dcaec10: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-0cbf2ae2189542948a3ffb6b88a4fa21 received from {username='slave'} at 127.0.0.1:51246
I20250831 07:58:38.774184 31893 tablet_copy_service.cc:434] P 8247a38b550649aa9cd6e7668dcaec10: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-0cbf2ae2189542948a3ffb6b88a4fa21 on tablet 0cbf2ae2189542948a3ffb6b88a4fa21 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:38.788765 32468 raft_consensus.cc:1215] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:38.910006 32468 raft_consensus.cc:1215] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:38.935566 32468 raft_consensus.cc:1215] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:38.990221 32468 raft_consensus.cc:1215] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:39.271216 347 raft_consensus.cc:1062] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:39.273998 347 consensus_queue.cc:237] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:39.287586 31881 raft_consensus.cc:1273] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:39.289260 32468 raft_consensus.cc:1273] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:39.290287 31728 raft_consensus.cc:1273] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:39.291350 348 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:39.297724 348 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:39.304558 425 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:39.327641 31881 raft_consensus.cc:2953] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.327384 428 raft_consensus.cc:2953] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.347301 32471 raft_consensus.cc:2953] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.363677 31630 catalog_manager.cc:5582] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.377848 31728 raft_consensus.cc:2953] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.436422 31804 consensus_queue.cc:237] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:39.442662 31728 raft_consensus.cc:1273] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:39.442885 32471 raft_consensus.cc:1273] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:39.444787 348 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:39.447955 428 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20250831 07:58:39.466864 428 raft_consensus.cc:2953] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.472638 32468 raft_consensus.cc:2953] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.476841 31617 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5eac48bce84e483582f863716d5c26f9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:39.480716 31728 raft_consensus.cc:2953] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.490603 31631 catalog_manager.cc:5582] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.508368 31873 tablet_service.cc:1515] Processing DeleteTablet for tablet 5eac48bce84e483582f863716d5c26f9 with delete_type TABLET_DATA_TOMBSTONED (TS 8247a38b550649aa9cd6e7668dcaec10 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57332
I20250831 07:58:39.521432 437 tablet_replica.cc:331] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10: stopping tablet replica
I20250831 07:58:39.523047 437 raft_consensus.cc:2241] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:39.538522 395 raft_consensus.cc:1062] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:39.542780 425 raft_consensus.cc:1062] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:39.542378 395 consensus_queue.cc:237] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:39.546782 425 consensus_queue.cc:237] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:39.555912 437 raft_consensus.cc:2270] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:39.560168 437 ts_tablet_manager.cc:1905] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:39.561923 31805 raft_consensus.cc:1273] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:39.564906 32755 raft_consensus.cc:1062] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:39.567674 389 consensus_queue.cc:1035] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:39.589787 32755 consensus_queue.cc:237] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:39.596454 437 ts_tablet_manager.cc:1918] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:39.596315 32468 raft_consensus.cc:1273] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:39.600780 437 log.cc:1199] T 5eac48bce84e483582f863716d5c26f9 P 8247a38b550649aa9cd6e7668dcaec10: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-2-root/wals/5eac48bce84e483582f863716d5c26f9
I20250831 07:58:39.603055 31727 raft_consensus.cc:1273] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:39.607502 387 consensus_queue.cc:1035] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20250831 07:58:39.625051 31616 catalog_manager.cc:4928] TS 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): tablet 5eac48bce84e483582f863716d5c26f9 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:39.625917 32468 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:39.628063 31881 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:39.626132 395 consensus_queue.cc:1035] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20250831 07:58:39.630409 428 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:39.632783 31727 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:39.634164 348 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:39.637229 425 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
W20250831 07:58:39.659003 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 54106.1us
I20250831 07:58:39.666745 395 raft_consensus.cc:2953] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.675542 32468 raft_consensus.cc:2953] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.682498 32467 raft_consensus.cc:2953] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.693889 32468 raft_consensus.cc:1273] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:39.695060 445 consensus_queue.cc:1035] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:39.696455 31725 raft_consensus.cc:1273] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:39.700762 441 consensus_queue.cc:1035] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:39.703261 31802 raft_consensus.cc:2953] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.702713 31727 raft_consensus.cc:2953] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.714978 31878 raft_consensus.cc:2953] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.727067 31801 raft_consensus.cc:1273] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:39.677795 425 raft_consensus.cc:2953] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.695739 31631 catalog_manager.cc:5582] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.739773 445 consensus_queue.cc:1035] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:39.774096 31881 consensus_queue.cc:237] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:39.785463 32471 raft_consensus.cc:1273] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:39.788621 395 consensus_queue.cc:1035] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:58:39.790676 31627 catalog_manager.cc:5582] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.798184 31801 raft_consensus.cc:1273] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:39.797860 31727 raft_consensus.cc:2953] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.804116 441 consensus_queue.cc:1035] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:39.818105 32755 raft_consensus.cc:2953] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.819413 32468 raft_consensus.cc:2953] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.821947 441 raft_consensus.cc:2953] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.826077 32467 raft_consensus.cc:2953] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.845263 31616 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 0cbf2ae2189542948a3ffb6b88a4fa21 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:39.845623 31725 raft_consensus.cc:2953] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.850060 31801 raft_consensus.cc:2953] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.860731 31630 catalog_manager.cc:5582] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.835727 31802 raft_consensus.cc:2953] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.887053 31804 consensus_queue.cc:237] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:39.896937 32471 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:39.896802 31630 catalog_manager.cc:5582] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.898602 31727 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:39.904204 454 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:39.908993 454 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:39.919510 31716 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cbf2ae2189542948a3ffb6b88a4fa21 with delete_type TABLET_DATA_TOMBSTONED (TS a0d845d01c8c4894895f66e3beae298c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57666
I20250831 07:58:39.919394 428 raft_consensus.cc:2953] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.936475 31725 raft_consensus.cc:2953] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:39.942308 32468 raft_consensus.cc:2953] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:40.082203 456 tablet_replica.cc:331] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c: stopping tablet replica
I20250831 07:58:40.083277 456 raft_consensus.cc:2241] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:40.083956 456 raft_consensus.cc:2270] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:40.097855 31878 consensus_queue.cc:237] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:40.100106 31628 catalog_manager.cc:5582] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:40.107393 31617 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 90effb264f51485dac220ee32f49e4ad with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250831 07:58:40.131227 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 177687us
I20250831 07:58:40.135414 32467 raft_consensus.cc:1273] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:40.136931 456 ts_tablet_manager.cc:1905] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:40.153726 31873 tablet_service.cc:1515] Processing DeleteTablet for tablet 90effb264f51485dac220ee32f49e4ad with delete_type TABLET_DATA_TOMBSTONED (TS 8247a38b550649aa9cd6e7668dcaec10 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57332
I20250831 07:58:40.156550 445 consensus_queue.cc:1035] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:40.185778 465 tablet_replica.cc:331] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10: stopping tablet replica
I20250831 07:58:40.186880 465 raft_consensus.cc:2241] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:40.187603 465 raft_consensus.cc:2270] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:40.188423 31728 raft_consensus.cc:1273] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:40.198885 32755 raft_consensus.cc:2953] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:40.205682 456 ts_tablet_manager.cc:1918] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:40.206221 456 log.cc:1199] T 0cbf2ae2189542948a3ffb6b88a4fa21 P a0d845d01c8c4894895f66e3beae298c: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-0-root/wals/0cbf2ae2189542948a3ffb6b88a4fa21
I20250831 07:58:40.208631 31618 catalog_manager.cc:4928] TS a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): tablet 0cbf2ae2189542948a3ffb6b88a4fa21 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:40.211038 465 ts_tablet_manager.cc:1905] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:40.210525 32467 raft_consensus.cc:2953] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:40.219849 31616 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a187e6c5ed8f4214b4486aab25955909 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:40.224921 31627 catalog_manager.cc:5582] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:40.229911 32755 consensus_queue.cc:1035] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:40.242756 31791 tablet_service.cc:1515] Processing DeleteTablet for tablet a187e6c5ed8f4214b4486aab25955909 with delete_type TABLET_DATA_TOMBSTONED (TS 7512ba2ed3a94982a22a75f51a647e65 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39068
I20250831 07:58:40.250221 465 ts_tablet_manager.cc:1918] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:40.250751 465 log.cc:1199] T 90effb264f51485dac220ee32f49e4ad P 8247a38b550649aa9cd6e7668dcaec10: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-2-root/wals/90effb264f51485dac220ee32f49e4ad
I20250831 07:58:40.252348 31616 catalog_manager.cc:4928] TS 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): tablet 90effb264f51485dac220ee32f49e4ad (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:40.261528 31727 raft_consensus.cc:2953] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:40.264470 466 tablet_replica.cc:331] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65: stopping tablet replica
I20250831 07:58:40.266603 466 raft_consensus.cc:2241] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:40.267489 466 raft_consensus.cc:2270] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:40.270725 466 ts_tablet_manager.cc:1905] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:40.290450 466 ts_tablet_manager.cc:1918] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:40.290870 466 log.cc:1199] T a187e6c5ed8f4214b4486aab25955909 P 7512ba2ed3a94982a22a75f51a647e65: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-1-root/wals/a187e6c5ed8f4214b4486aab25955909
I20250831 07:58:40.292502 31617 catalog_manager.cc:4928] TS 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): tablet a187e6c5ed8f4214b4486aab25955909 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:41.308344 31882 consensus_queue.cc:237] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:41.315475 31801 raft_consensus.cc:1273] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:41.316466 31727 raft_consensus.cc:1273] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:41.316767 387 consensus_queue.cc:1035] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:58:41.317739 31847 consensus_peers.cc:489] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26525016efb247c3b1c2bf7fec0e1658. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:41.317798 32755 consensus_queue.cc:1035] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:41.326428 387 raft_consensus.cc:2953] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:41.331593 31727 raft_consensus.cc:2953] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:41.334313 31800 raft_consensus.cc:2953] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:41.356707 31627 catalog_manager.cc:5582] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:41.353253 31728 consensus_queue.cc:237] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
W20250831 07:58:41.395525 31698 consensus_peers.cc:489] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3d650c422ee41af95705f3fd876ddb8. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:41.396831 31881 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:41.396831 31801 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:41.409047 459 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:41.412225 464 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:41.433748 31882 raft_consensus.cc:2953] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:41.430318 464 raft_consensus.cc:2953] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:41.486723 31801 consensus_queue.cc:237] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
W20250831 07:58:41.470791 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 61049.5us
I20250831 07:58:41.523919 31878 raft_consensus.cc:1273] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250831 07:58:41.526554 31773 consensus_peers.cc:489] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d77811606c643a4970bc6b8a5f39fdb. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:41.527642 461 consensus_queue.cc:1035] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:41.535619 31727 raft_consensus.cc:1273] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:41.527230 31628 catalog_manager.cc:5582] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:41.540040 458 consensus_queue.cc:1035] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:58:41.581532 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 70727.9us
I20250831 07:58:41.581442 461 raft_consensus.cc:2953] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:41.592981 31882 raft_consensus.cc:2953] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:41.599288 31802 raft_consensus.cc:2953] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:41.609225 31878 consensus_queue.cc:237] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:41.617592 31727 raft_consensus.cc:2953] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:41.615679 31628 catalog_manager.cc:5582] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250831 07:58:41.629964 31847 consensus_peers.cc:489] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 -> Peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319): Couldn't send request to peer 7ecb257174554a26b2c35ad04b0eabae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9949d6ea1bea439fa2ca1fff1c5641bd. This is attempt 1: this message will repeat every 5th retry.
I20250831 07:58:41.631373 31802 raft_consensus.cc:1273] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:41.633873 32755 consensus_queue.cc:1035] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:41.634654 31725 raft_consensus.cc:1273] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 07:58:41.636262 32755 consensus_queue.cc:1035] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:41.659641 467 raft_consensus.cc:2953] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:41.669771 31725 raft_consensus.cc:2953] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:41.672559 31802 raft_consensus.cc:2953] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } } }
I20250831 07:58:41.689356 31628 catalog_manager.cc:5582] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index -1 to 2, NON_VOTER 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) added. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:41.901671 487 ts_tablet_manager.cc:927] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:58:41.905689 487 tablet_copy_client.cc:323] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.1:45027
I20250831 07:58:41.907516 31744 tablet_copy_service.cc:140] P a0d845d01c8c4894895f66e3beae298c: Received BeginTabletCopySession request for tablet d3d650c422ee41af95705f3fd876ddb8 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43230)
I20250831 07:58:41.908034 31744 tablet_copy_service.cc:161] P a0d845d01c8c4894895f66e3beae298c: Beginning new tablet copy session on tablet d3d650c422ee41af95705f3fd876ddb8 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43230: session id = 7ecb257174554a26b2c35ad04b0eabae-d3d650c422ee41af95705f3fd876ddb8
I20250831 07:58:41.914678 31744 tablet_copy_source_session.cc:215] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:41.918184 487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3d650c422ee41af95705f3fd876ddb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:41.944150 489 ts_tablet_manager.cc:927] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:41.944293 487 tablet_copy_client.cc:806] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:41.945152 487 tablet_copy_client.cc:670] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:41.951346 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:41.955926 487 tablet_copy_client.cc:538] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:41.984130 489 tablet_copy_client.cc:323] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.3:42585
I20250831 07:58:41.988332 31893 tablet_copy_service.cc:140] P 8247a38b550649aa9cd6e7668dcaec10: Received BeginTabletCopySession request for tablet 26525016efb247c3b1c2bf7fec0e1658 from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:51246)
I20250831 07:58:41.988946 31893 tablet_copy_service.cc:161] P 8247a38b550649aa9cd6e7668dcaec10: Beginning new tablet copy session on tablet 26525016efb247c3b1c2bf7fec0e1658 from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:51246: session id = 7ecb257174554a26b2c35ad04b0eabae-26525016efb247c3b1c2bf7fec0e1658
I20250831 07:58:41.989145 487 tablet_bootstrap.cc:492] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:41.999521 31893 tablet_copy_source_session.cc:215] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:42.058315 489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26525016efb247c3b1c2bf7fec0e1658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:42.072006 487 tablet_bootstrap.cc:492] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:42.073405 487 tablet_bootstrap.cc:492] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:42.074318 487 ts_tablet_manager.cc:1397] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.085s user 0.015s sys 0.012s
I20250831 07:58:42.081039 487 raft_consensus.cc:357] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:42.082131 487 raft_consensus.cc:738] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:42.083105 491 ts_tablet_manager.cc:927] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:42.082805 487 consensus_queue.cc:260] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:42.098253 487 ts_tablet_manager.cc:1428] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.024s user 0.007s sys 0.007s
I20250831 07:58:42.100121 31744 tablet_copy_service.cc:342] P a0d845d01c8c4894895f66e3beae298c: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-d3d650c422ee41af95705f3fd876ddb8 received from {username='slave'} at 127.0.0.1:43230
I20250831 07:58:42.100574 31744 tablet_copy_service.cc:434] P a0d845d01c8c4894895f66e3beae298c: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-d3d650c422ee41af95705f3fd876ddb8 on tablet d3d650c422ee41af95705f3fd876ddb8 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:42.103713 491 tablet_copy_client.cc:323] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.2:37261
I20250831 07:58:42.105454 31725 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643"
dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:48724
I20250831 07:58:42.106233 31725 raft_consensus.cc:604] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:42.109591 31819 tablet_copy_service.cc:140] P 7512ba2ed3a94982a22a75f51a647e65: Received BeginTabletCopySession request for tablet 6d77811606c643a4970bc6b8a5f39fdb from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:43428)
I20250831 07:58:42.110174 31819 tablet_copy_service.cc:161] P 7512ba2ed3a94982a22a75f51a647e65: Beginning new tablet copy session on tablet 6d77811606c643a4970bc6b8a5f39fdb from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:43428: session id = 7ecb257174554a26b2c35ad04b0eabae-6d77811606c643a4970bc6b8a5f39fdb
I20250831 07:58:42.129371 31819 tablet_copy_source_session.cc:215] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:42.131414 487 ts_tablet_manager.cc:927] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: Initiating tablet copy from peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:42.135239 491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d77811606c643a4970bc6b8a5f39fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:42.138724 487 tablet_copy_client.cc:323] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Beginning tablet copy session from remote peer at address 127.29.217.3:42585
I20250831 07:58:42.141022 31893 tablet_copy_service.cc:140] P 8247a38b550649aa9cd6e7668dcaec10: Received BeginTabletCopySession request for tablet 9949d6ea1bea439fa2ca1fff1c5641bd from peer 7ecb257174554a26b2c35ad04b0eabae ({username='slave'} at 127.0.0.1:51246)
I20250831 07:58:42.141636 31893 tablet_copy_service.cc:161] P 8247a38b550649aa9cd6e7668dcaec10: Beginning new tablet copy session on tablet 9949d6ea1bea439fa2ca1fff1c5641bd from peer 7ecb257174554a26b2c35ad04b0eabae at {username='slave'} at 127.0.0.1:51246: session id = 7ecb257174554a26b2c35ad04b0eabae-9949d6ea1bea439fa2ca1fff1c5641bd
I20250831 07:58:42.180505 31893 tablet_copy_source_session.cc:215] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 07:58:42.183981 487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9949d6ea1bea439fa2ca1fff1c5641bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 07:58:42.200435 491 tablet_copy_client.cc:806] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:42.201411 491 tablet_copy_client.cc:670] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:42.203291 31883 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:58:42.203969 31883 raft_consensus.cc:604] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:42.204918 489 tablet_copy_client.cc:806] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:42.205546 489 tablet_copy_client.cc:670] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:42.207995 31878 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:58:42.208581 31878 raft_consensus.cc:604] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:42.210669 31878 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:58:42.211133 31878 raft_consensus.cc:604] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:42.213311 31883 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:58:42.213887 31883 raft_consensus.cc:604] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:42.217617 31878 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:58:42.218047 491 tablet_copy_client.cc:538] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:42.218132 31878 raft_consensus.cc:604] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:42.219349 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250831 07:58:42.220610 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:58:42.221381 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:42.221947 487 tablet_copy_client.cc:806] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 0 data blocks...
I20250831 07:58:42.222518 487 tablet_copy_client.cc:670] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Starting download of 1 WAL segments...
I20250831 07:58:42.226882 489 tablet_copy_client.cc:538] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:42.232708 487 tablet_copy_client.cc:538] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 07:58:42.249469 491 tablet_bootstrap.cc:492] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:42.255177 487 tablet_bootstrap.cc:492] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:42.287603 487 tablet_bootstrap.cc:492] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:42.292414 487 tablet_bootstrap.cc:492] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:42.293227 487 ts_tablet_manager.cc:1397] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.038s user 0.024s sys 0.003s
I20250831 07:58:42.297343 487 raft_consensus.cc:357] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:42.298288 487 raft_consensus.cc:738] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:42.251744 489 tablet_bootstrap.cc:492] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap starting.
I20250831 07:58:42.299023 487 consensus_queue.cc:260] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:42.304474 501 raft_consensus.cc:991] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c: : Instructing follower 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:58:42.306815 501 raft_consensus.cc:1079] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Signalling peer 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:58:42.306857 491 tablet_bootstrap.cc:492] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:42.309759 31802 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:39084
I20250831 07:58:42.310496 31802 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:42.311115 31802 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.315716 32755 raft_consensus.cc:991] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10: : Instructing follower 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:58:42.316243 32755 raft_consensus.cc:1079] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Signalling peer 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:58:42.318234 31800 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:39082
I20250831 07:58:42.318799 31800 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:42.318964 387 raft_consensus.cc:991] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10: : Instructing follower 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:58:42.319180 31800 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.319411 387 raft_consensus.cc:1079] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Signalling peer 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:58:42.321508 31801 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:39082
I20250831 07:58:42.322005 31801 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:42.322368 31801 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.326610 31800 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:42.310052 487 ts_tablet_manager.cc:1428] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.016s user 0.003s sys 0.003s
I20250831 07:58:42.331089 31802 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:42.333484 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:42.334177 31878 raft_consensus.cc:3053] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:42.334471 31878 raft_consensus.cc:738] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:42.335166 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:42.335068 31878 consensus_queue.cc:260] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:42.335628 31728 raft_consensus.cc:3053] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:42.335907 31728 raft_consensus.cc:738] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:42.336202 31878 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.336470 31728 consensus_queue.cc:260] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:42.337224 31728 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.342433 491 tablet_bootstrap.cc:492] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:42.344262 491 ts_tablet_manager.cc:1397] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.095s user 0.023s sys 0.017s
I20250831 07:58:42.331264 31801 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:42.342335 31893 tablet_copy_service.cc:342] P 8247a38b550649aa9cd6e7668dcaec10: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-9949d6ea1bea439fa2ca1fff1c5641bd received from {username='slave'} at 127.0.0.1:51246
I20250831 07:58:42.470002 31801 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:42.348011 31800 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:58:42.472872 31893 tablet_copy_service.cc:434] P 8247a38b550649aa9cd6e7668dcaec10: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-9949d6ea1bea439fa2ca1fff1c5641bd on tablet 9949d6ea1bea439fa2ca1fff1c5641bd with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:42.349953 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:42.474632 31725 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.349279 31728 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
W20250831 07:58:42.474723 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 85105.5us
I20250831 07:58:42.350469 31802 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:42.375931 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:42.480144 31882 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.480639 32468 raft_consensus.cc:1215] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:42.486781 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:42.488911 31775 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:42.493837 31725 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:42.499485 482 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:42.500068 482 raft_consensus.cc:695] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:42.499928 491 raft_consensus.cc:357] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:42.500876 491 raft_consensus.cc:738] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:42.501034 482 consensus_queue.cc:237] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:42.501556 491 consensus_queue.cc:260] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:42.507915 31728 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.509455 31775 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:42.510696 458 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:42.513877 32468 raft_consensus.cc:1215] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:42.349068 31878 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:42.519809 491 ts_tablet_manager.cc:1428] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.046s user 0.006s sys 0.000s
I20250831 07:58:42.535763 31819 tablet_copy_service.cc:342] P 7512ba2ed3a94982a22a75f51a647e65: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-6d77811606c643a4970bc6b8a5f39fdb received from {username='slave'} at 127.0.0.1:43428
I20250831 07:58:42.536271 31819 tablet_copy_service.cc:434] P 7512ba2ed3a94982a22a75f51a647e65: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-6d77811606c643a4970bc6b8a5f39fdb on tablet 6d77811606c643a4970bc6b8a5f39fdb with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:42.526780 31728 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:42.544452 31882 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:42.545341 31775 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:58:42.545499 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 161699us
I20250831 07:58:42.546391 511 raft_consensus.cc:2802] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:42.547350 511 raft_consensus.cc:695] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:42.548588 511 consensus_queue.cc:237] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:42.582186 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:42.582826 31878 raft_consensus.cc:3053] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:42.583117 31878 raft_consensus.cc:738] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:42.588413 458 raft_consensus.cc:695] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:42.583803 31878 consensus_queue.cc:260] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:42.590540 31878 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.613152 31878 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:42.589375 458 consensus_queue.cc:237] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:42.626349 489 tablet_bootstrap.cc:492] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 07:58:42.630605 489 tablet_bootstrap.cc:492] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae: Bootstrap complete.
I20250831 07:58:42.631297 489 ts_tablet_manager.cc:1397] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae: Time spent bootstrapping tablet: real 0.380s user 0.030s sys 0.035s
I20250831 07:58:42.634290 489 raft_consensus.cc:357] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:42.636510 31627 catalog_manager.cc:5582] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 2, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:42.653100 489 raft_consensus.cc:738] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Initialized, Role: LEARNER
I20250831 07:58:42.654803 489 consensus_queue.cc:260] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:42.659490 506 raft_consensus.cc:991] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10: : Instructing follower 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:58:42.660269 32755 raft_consensus.cc:1079] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Signalling peer 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:58:42.659464 489 ts_tablet_manager.cc:1428] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20250831 07:58:42.670562 31805 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:39082
I20250831 07:58:42.671134 31805 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:42.671514 31805 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.676473 31893 tablet_copy_service.cc:342] P 8247a38b550649aa9cd6e7668dcaec10: Request end of tablet copy session 7ecb257174554a26b2c35ad04b0eabae-26525016efb247c3b1c2bf7fec0e1658 received from {username='slave'} at 127.0.0.1:51246
I20250831 07:58:42.676916 31893 tablet_copy_service.cc:434] P 8247a38b550649aa9cd6e7668dcaec10: ending tablet copy session 7ecb257174554a26b2c35ad04b0eabae-26525016efb247c3b1c2bf7fec0e1658 on tablet 26525016efb247c3b1c2bf7fec0e1658 with peer 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:42.678354 31805 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:42.680578 31805 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:42.681293 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:42.681843 31725 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.688493 31725 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:42.689723 31775 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:42.692239 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:42.692761 31881 raft_consensus.cc:3053] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:42.692953 458 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:42.693051 31881 raft_consensus.cc:738] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:42.693532 458 raft_consensus.cc:695] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:42.693905 31881 consensus_queue.cc:260] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:42.694412 458 consensus_queue.cc:237] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:42.695133 31881 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.711378 507 raft_consensus.cc:991] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10: : Instructing follower 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:42.711480 31881 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:42.712024 507 raft_consensus.cc:1079] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Signalling peer 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:42.715072 32467 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3"
dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:47886
I20250831 07:58:42.715590 32467 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:42.715878 32467 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.721817 32467 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:42.726444 32467 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:42.728986 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:42.729539 31805 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.735055 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:42.735741 31881 raft_consensus.cc:3053] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:42.736091 31881 raft_consensus.cc:738] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:42.736816 31881 consensus_queue.cc:260] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:42.737372 31805 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:42.738178 31881 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.738766 32435 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:42.739631 31627 catalog_manager.cc:5582] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 2, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:42.741320 496 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:42.741145 31627 catalog_manager.cc:5582] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 2, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:42.742545 31627 catalog_manager.cc:5582] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 2, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:42.788525 31881 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:42.798537 496 raft_consensus.cc:695] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:42.799630 496 consensus_queue.cc:237] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:42.816057 506 raft_consensus.cc:991] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10: : Instructing follower 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:42.816588 506 raft_consensus.cc:1079] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Signalling peer 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:42.817993 32471 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed"
dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:47886
I20250831 07:58:42.818493 32471 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:42.818820 32471 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.831571 32471 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:42.835556 32471 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:42.836619 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:42.837242 31881 raft_consensus.cc:3053] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:42.837534 31881 raft_consensus.cc:738] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:42.838227 31881 consensus_queue.cc:260] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:42.842384 31881 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.847617 31627 catalog_manager.cc:5582] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 2, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 2 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:42.854108 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:42.854688 31725 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:42.862273 31725 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:42.863548 32436 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:42.866271 496 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:42.866968 467 raft_consensus.cc:1062] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:42.867770 496 raft_consensus.cc:695] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:42.868257 31881 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:42.869020 496 consensus_queue.cc:237] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:42.869586 467 consensus_queue.cc:237] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:42.887166 31725 raft_consensus.cc:1273] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:42.887200 32471 raft_consensus.cc:1273] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:42.888657 31627 catalog_manager.cc:5582] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 2, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 2 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:42.889819 467 consensus_queue.cc:1035] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:42.891032 31807 raft_consensus.cc:1273] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:42.892371 508 consensus_queue.cc:1035] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:42.895969 508 consensus_queue.cc:1035] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:42.913779 506 raft_consensus.cc:2953] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:42.920845 31728 raft_consensus.cc:2953] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:42.917351 32471 raft_consensus.cc:2953] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:42.928122 31807 raft_consensus.cc:2953] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:42.950053 31627 catalog_manager.cc:5582] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } attrs { replace: true } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:42.969174 500 raft_consensus.cc:1062] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:42.975943 500 consensus_queue.cc:237] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:42.991755 31882 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:42.992045 31802 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:42.992831 32471 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:42.994001 500 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:42.995502 504 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:42.999672 475 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:43.007915 31878 consensus_queue.cc:237] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:43.016743 32467 raft_consensus.cc:1215] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:43.017064 32468 raft_consensus.cc:1273] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:43.017616 31725 raft_consensus.cc:1273] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:43.018862 507 consensus_queue.cc:1035] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:43.021647 507 consensus_queue.cc:1035] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:58:43.030908 32755 raft_consensus.cc:2953] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:43.041422 31878 raft_consensus.cc:1273] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:43.042881 512 consensus_queue.cc:1035] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:43.045301 31616 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9949d6ea1bea439fa2ca1fff1c5641bd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:43.047390 32468 raft_consensus.cc:2953] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:43.064739 31725 raft_consensus.cc:2953] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:43.073840 31627 catalog_manager.cc:5582] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: config changed from index 3 to 4, VOTER 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) evicted. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:43.076483 32467 raft_consensus.cc:1215] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 07:58:43.080716 31728 raft_consensus.cc:1273] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:43.082167 482 consensus_queue.cc:1035] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:43.117240 31730 raft_consensus.cc:1273] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:43.117684 504 raft_consensus.cc:2953] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:43.118891 482 consensus_queue.cc:1035] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:43.129798 32470 raft_consensus.cc:2953] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:43.143312 31791 tablet_service.cc:1515] Processing DeleteTablet for tablet 9949d6ea1bea439fa2ca1fff1c5641bd with delete_type TABLET_DATA_TOMBSTONED (TS 7512ba2ed3a94982a22a75f51a647e65 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39068
I20250831 07:58:43.152498 31878 raft_consensus.cc:1273] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:43.158367 31878 raft_consensus.cc:2953] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:43.184978 31733 raft_consensus.cc:1273] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:43.188431 458 consensus_queue.cc:1035] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:43.195040 533 tablet_replica.cc:331] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65: stopping tablet replica
I20250831 07:58:43.196522 533 raft_consensus.cc:2241] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:43.197476 533 raft_consensus.cc:2270] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:43.201114 533 ts_tablet_manager.cc:1905] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250831 07:58:43.206974 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 50893us
I20250831 07:58:43.210304 461 consensus_queue.cc:1035] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:43.228639 31882 raft_consensus.cc:1273] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:43.230504 31879 raft_consensus.cc:1273] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:43.243341 533 ts_tablet_manager.cc:1918] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:43.256095 533 log.cc:1199] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7512ba2ed3a94982a22a75f51a647e65: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-1-root/wals/9949d6ea1bea439fa2ca1fff1c5641bd
I20250831 07:58:43.254570 31631 catalog_manager.cc:5582] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:43.258340 31617 catalog_manager.cc:4928] TS 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): tablet 9949d6ea1bea439fa2ca1fff1c5641bd (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:43.300045 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:43.351526 31879 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:43.321472 512 consensus_queue.cc:1035] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20250831 07:58:43.325803 458 consensus_queue.cc:1035] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
W20250831 07:58:43.348397 539 log.cc:927] Time spent T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.067s user 0.002s sys 0.000s
I20250831 07:58:43.366771 496 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250831 07:58:43.367945 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 46066.3us
W20250831 07:58:43.380760 531 log.cc:927] Time spent T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20250831 07:58:43.381700 31878 raft_consensus.cc:1273] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250831 07:58:43.404414 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 76162.6us
I20250831 07:58:43.429764 31802 raft_consensus.cc:1273] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:43.438967 496 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250831 07:58:43.447433 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 59759.4us
I20250831 07:58:43.459769 496 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:58:43.470398 31728 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:43.482039 31728 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb"
dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:48724
I20250831 07:58:43.482656 31728 raft_consensus.cc:604] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
W20250831 07:58:43.488197 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 98295.1us
I20250831 07:58:43.487097 31730 consensus_queue.cc:237] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:43.496208 31728 raft_consensus.cc:1273] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:43.554600 496 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:58:43.570775 31804 raft_consensus.cc:2953] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:43.596829 32468 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:43.576083 547 consensus_queue.cc:1035] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20250831 07:58:43.655536 31802 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:58:43.664577 501 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20250831 07:58:43.729753 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 124815us
W20250831 07:58:43.740558 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 163961us
W20250831 07:58:43.762805 555 log.cc:927] Time spent T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.076s user 0.002s sys 0.000s
W20250831 07:58:43.778889 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 184186us
I20250831 07:58:43.802493 557 raft_consensus.cc:991] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: : Instructing follower 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:58:43.828120 566 raft_consensus.cc:1079] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Signalling peer 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:58:43.831902 480 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
W20250831 07:58:43.894940 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 145126us
W20250831 07:58:43.952710 530 log.cc:927] Time spent T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20250831 07:58:43.962877 549 raft_consensus.cc:1062] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:43.965453 549 consensus_queue.cc:237] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:43.988098 500 raft_consensus.cc:2953] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:43.989413 32468 raft_consensus.cc:2953] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
W20250831 07:58:44.012506 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:44.105093 31806 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:39084
I20250831 07:58:44.105696 31806 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:44.106099 31806 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:44.155757 31806 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:44.229553 31806 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:44.261235 31618 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d3d650c422ee41af95705f3fd876ddb8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:58:44.271802 31806 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:37044
I20250831 07:58:44.272646 31806 raft_consensus.cc:604] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:58:44.283213 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 441911us
I20250831 07:58:44.334049 31879 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:58:44.334661 31879 raft_consensus.cc:604] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
W20250831 07:58:44.341939 583 log.cc:927] Time spent T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20250831 07:58:44.362321 31806 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:37044
I20250831 07:58:44.365326 31806 raft_consensus.cc:604] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:44.366281 31805 raft_consensus.cc:2953] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:44.395066 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:44.398815 31734 raft_consensus.cc:3053] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:44.402570 31802 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
mode: GRACEFUL
new_leader_uuid: "8247a38b550649aa9cd6e7668dcaec10"
from {username='slave'} at 127.0.0.1:37044
I20250831 07:58:44.408128 31802 raft_consensus.cc:604] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Received request to transfer leadership to TS 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:44.407689 31734 raft_consensus.cc:738] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:44.423341 31734 consensus_queue.cc:260] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:44.508149 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 84935.7us
W20250831 07:58:44.424113 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 315390us
I20250831 07:58:44.515944 31734 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:58:44.517470 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 84972.8us
W20250831 07:58:44.517125 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 137504us
W20250831 07:58:44.578011 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 59102.6us
W20250831 07:58:44.583894 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 64504.3us
I20250831 07:58:44.611477 508 raft_consensus.cc:1062] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10: attempting to promote NON_VOTER 7ecb257174554a26b2c35ad04b0eabae to VOTER
I20250831 07:58:44.625766 31805 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:37044
I20250831 07:58:44.626236 31879 raft_consensus.cc:1273] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:44.626418 31805 raft_consensus.cc:604] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:44.626502 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:44.627076 31883 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:44.641366 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
W20250831 07:58:44.642781 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 262171us
I20250831 07:58:44.684885 31734 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:44.692283 31883 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:44.698168 575 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:44.698944 575 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:44.740778 32471 raft_consensus.cc:1273] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250831 07:58:44.746915 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 157611us
I20250831 07:58:44.745242 588 raft_consensus.cc:991] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: : Instructing follower 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:44.769973 575 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:44.771667 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:44.774014 596 raft_consensus.cc:991] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65: : Instructing follower 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:44.758013 549 raft_consensus.cc:1079] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Signalling peer 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:44.774541 596 raft_consensus.cc:1079] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Signalling peer 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:44.776105 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:44.777982 31848 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:44.779428 31725 raft_consensus.cc:1273] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:44.780989 580 raft_consensus.cc:2747] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:44.784672 31775 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:44.785985 546 raft_consensus.cc:991] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10: : Instructing follower 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:58:44.786980 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:44.831771 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 43646.1us
W20250831 07:58:44.831918 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 65061.4us
I20250831 07:58:44.839529 602 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:44.831791 546 raft_consensus.cc:1079] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Signalling peer 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:58:44.856240 602 raft_consensus.cc:695] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:44.857290 602 consensus_queue.cc:237] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:44.861121 540 raft_consensus.cc:991] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65: : Instructing follower 8247a38b550649aa9cd6e7668dcaec10 to start an election
I20250831 07:58:44.840030 549 consensus_queue.cc:1035] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20250831 07:58:44.870553 543 consensus_queue.cc:1035] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:44.871714 588 consensus_queue.cc:1035] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:44.873208 31806 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:39082
I20250831 07:58:44.874745 31806 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:44.875098 31806 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:44.877882 32468 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad"
dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:38362
I20250831 07:58:44.878407 32471 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0"
dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:38362
I20250831 07:58:44.878917 32471 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:44.879285 32471 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:44.872047 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 92521.8us
I20250831 07:58:44.873924 564 raft_consensus.cc:1079] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Signalling peer 8247a38b550649aa9cd6e7668dcaec10 to start an election
I20250831 07:58:44.878420 32468 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:44.884977 32468 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:44.886184 31631 catalog_manager.cc:5582] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 3 to 4, VOTER 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) evicted. New cstate: current_term: 1 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:44.889175 31806 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:44.890509 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:58:44.891276 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:44.894755 32468 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:44.899174 32471 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:44.902211 31878 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
from {username='slave'} at 127.0.0.1:57340
I20250831 07:58:44.902788 31878 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:44.903136 31878 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:44.905114 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:44.905687 31802 raft_consensus.cc:3053] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:44.906000 31802 raft_consensus.cc:738] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:44.906683 31802 consensus_queue.cc:260] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:44.907701 31802 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:44.908157 31806 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:44.908492 32471 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:44.910117 31878 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:44.913012 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:44.913559 31728 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:44.921669 31878 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:44.923008 564 raft_consensus.cc:991] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65: : Instructing follower 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:44.935639 564 raft_consensus.cc:1079] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Signalling peer 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:44.941082 32470 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a"
dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:38362
I20250831 07:58:44.941602 32470 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:44.941911 32470 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:44.930828 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:44.931529 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:44.931743 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:44.932459 31728 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:44.944141 32468 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:44.944849 31727 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:44.945173 32467 raft_consensus.cc:1238] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:44.945065 31804 raft_consensus.cc:3053] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:44.946888 31775 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:44.956151 31804 raft_consensus.cc:738] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:44.945590 31881 raft_consensus.cc:3053] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:44.957531 31881 raft_consensus.cc:738] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:44.934150 31802 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:44.934860 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:44.960599 31805 raft_consensus.cc:3053] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:44.960894 31805 raft_consensus.cc:738] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:44.961723 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:44.962293 31728 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:44.961552 31805 consensus_queue.cc:260] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:44.965586 31881 consensus_queue.cc:260] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:44.965293 31804 consensus_queue.cc:260] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:44.968319 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:44.968865 31730 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:44.969584 31804 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:44.970345 584 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:44.970728 32470 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:44.971640 32471 raft_consensus.cc:1238] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:44.974172 32435 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:44.975476 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:44.975929 31806 raft_consensus.cc:3053] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:44.976281 31806 raft_consensus.cc:738] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:44.976130 577 consensus_queue.cc:1046] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20250831 07:58:44.977164 32470 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:44.976994 31806 consensus_queue.cc:260] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:44.978073 31806 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:44.981016 508 consensus_queue.cc:237] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:44.985644 31727 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:44.987015 31806 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:44.988359 32435 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:44.990887 519 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:44.991047 31728 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:44.993432 32436 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:44.995035 31881 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:45.001921 31881 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:45.003392 31805 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:45.006788 31804 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:44.973412 584 raft_consensus.cc:695] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:45.008472 496 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:45.009119 496 raft_consensus.cc:695] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:45.010138 496 consensus_queue.cc:237] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:45.017416 31848 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:45.024220 507 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:45.024827 507 raft_consensus.cc:695] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:45.025718 507 consensus_queue.cc:237] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:45.027554 545 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:45.028144 545 raft_consensus.cc:695] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:45.017581 584 consensus_queue.cc:237] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:45.029036 545 consensus_queue.cc:237] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:45.017374 519 raft_consensus.cc:695] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:45.078135 31730 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
W20250831 07:58:45.110204 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 121909us
I20250831 07:58:45.200656 31805 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:45.278328 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:45.279138 31881 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:45.289979 31881 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
W20250831 07:58:45.208384 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 215013us
I20250831 07:58:45.326251 31802 raft_consensus.cc:1238] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:45.311482 602 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:45.317986 519 consensus_queue.cc:237] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:45.380851 32755 raft_consensus.cc:1025] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: attempt to promote peer 7ecb257174554a26b2c35ad04b0eabae: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20250831 07:58:45.509492 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 97171.1us
I20250831 07:58:45.356901 602 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:45.514447 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 146230us
W20250831 07:58:45.590173 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 187663us
W20250831 07:58:45.610972 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 96550.2us
W20250831 07:58:45.628235 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 117573us
W20250831 07:58:45.628748 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 58454.6us
W20250831 07:58:45.628261 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:45.512750 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 80674.9us
W20250831 07:58:45.643558 617 log.cc:927] Time spent T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.222s user 0.000s sys 0.000s
W20250831 07:58:45.666335 616 log.cc:927] Time spent T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.280s user 0.001s sys 0.001s
W20250831 07:58:45.676625 618 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975: Append to log took a long time: real 0.247s user 0.000s sys 0.001s
I20250831 07:58:45.681320 32469 raft_consensus.cc:1273] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 LEARNER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:45.688853 602 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:45.689162 541 consensus_queue.cc:1035] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:45.696106 31730 raft_consensus.cc:1273] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:45.702139 31733 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:45.704001 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:58:45.695329 615 log.cc:927] Time spent T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.261s user 0.001s sys 0.000s
I20250831 07:58:45.729795 31733 raft_consensus.cc:1273] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:45.740689 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:45.720113 554 consensus_queue.cc:1035] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20250831 07:58:45.749469 626 consensus_queue.cc:1035] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20250831 07:58:45.732223 496 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20250831 07:58:45.749758 31802 raft_consensus.cc:1273] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250831 07:58:45.774197 620 log.cc:927] Time spent T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20250831 07:58:45.773382 588 raft_consensus.cc:2953] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:45.813933 31877 raft_consensus.cc:1273] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250831 07:58:45.828090 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 47726.7us
I20250831 07:58:45.829620 31725 raft_consensus.cc:1273] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:58:45.833220 624 consensus_queue.cc:1035] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20250831 07:58:45.832751 31773 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:45.838208 31881 raft_consensus.cc:1273] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:58:45.841843 590 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:45.852178 601 consensus_queue.cc:1035] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20250831 07:58:45.861205 31881 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:45.866142 605 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:45.866496 31805 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:45.868229 31805 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:45.866729 605 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: NON_VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: true } }
I20250831 07:58:45.869964 31805 raft_consensus.cc:1273] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:45.862638 496 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20250831 07:58:45.871547 540 consensus_queue.cc:1035] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
W20250831 07:58:45.948879 618 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20250831 07:58:45.946941 560 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:45.959028 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:45.965382 610 log.cc:927] Time spent T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.555s user 0.000s sys 0.004s
I20250831 07:58:45.966377 560 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:45.969911 543 consensus_queue.cc:1035] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20250831 07:58:45.971590 32471 raft_consensus.cc:2953] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:45.977535 560 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:45.956430 605 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:45.983786 31734 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:45.986882 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:45.991163 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:46.028846 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 46823.6us
I20250831 07:58:46.046530 31699 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:46.053870 519 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20250831 07:58:46.079308 31878 raft_consensus.cc:2953] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:46.034094 31734 raft_consensus.cc:1273] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 07:58:46.032299 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:46.102257 613 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
W20250831 07:58:46.136718 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 90284.5us
I20250831 07:58:46.143383 475 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:46.144129 507 consensus_queue.cc:1035] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:46.061169 571 consensus_queue.cc:1035] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.095s
I20250831 07:58:46.157763 31699 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:46.158713 31727 raft_consensus.cc:2953] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:46.172279 31631 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43950 (request call id 118) took 2049 ms. Trace:
W20250831 07:58:46.245483 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 64966us
I20250831 07:58:46.250156 31800 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:46.253873 31873 tablet_service.cc:1515] Processing DeleteTablet for tablet d3d650c422ee41af95705f3fd876ddb8 with delete_type TABLET_DATA_TOMBSTONED (TS 8247a38b550649aa9cd6e7668dcaec10 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57332
W20250831 07:58:46.253782 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 194250us
I20250831 07:58:46.172722 31631 rpcz_store.cc:276] 0831 07:58:44.109077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:44.109291 (+ 214us) service_pool.cc:225] Handling call
0831 07:58:46.158336 (+2049045us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:58:44.944591 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:58:44.945125 (+ 534us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:58:44.945157 (+ 32us) write_op.cc:435] Acquired schema lock
0831 07:58:44.945170 (+ 13us) tablet.cc:660] Decoding operations
0831 07:58:44.945470 (+ 300us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:58:44.964903 (+ 19433us) write_op.cc:641] Partition lock acquired for write op
0831 07:58:44.967234 (+ 2331us) tablet.cc:683] Acquiring locks for 1 operations
0831 07:58:45.005048 (+ 37814us) tablet.cc:699] Row locks acquired
0831 07:58:45.013347 (+ 8299us) write_op.cc:260] PREPARE: finished
0831 07:58:45.020361 (+ 7014us) write_op.cc:270] Start()
0831 07:58:45.044224 (+ 23863us) write_op.cc:276] Timestamp: P: 1756627125020007 usec, L: 0
0831 07:58:45.044249 (+ 25us) op_driver.cc:348] REPLICATION: starting
0831 07:58:45.263392 (+219143us) log.cc:844] Serialized 533 byte log entry
0831 07:58:45.751264 (+487872us) op_driver.cc:464] REPLICATION: finished
0831 07:58:45.859077 (+107813us) write_op.cc:301] APPLY: starting
0831 07:58:45.859159 (+ 82us) tablet.cc:1365] starting BulkCheckPresence
0831 07:58:45.859264 (+ 105us) tablet.cc:1368] finished BulkCheckPresence
0831 07:58:45.859471 (+ 207us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:58:45.859682 (+ 211us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:58:45.859868 (+ 186us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0831 07:58:45.859919 (+ 51us) write_op.cc:312] APPLY: finished
0831 07:58:45.882215 (+ 22296us) log.cc:844] Serialized 23 byte log entry
0831 07:58:45.891469 (+ 9254us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:45.900157 (+ 8688us) write_op.cc:454] Released schema lock
0831 07:58:45.927121 (+ 26964us) write_op.cc:341] FINISH: Updating metrics
0831 07:58:45.947665 (+ 20544us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:45.949933 (+ 2268us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":496882,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":107572,"apply.run_cpu_time_us":2155,"apply.run_wall_time_us":87062,"num_ops":1,"prepare.queue_time_us":55851,"prepare.run_cpu_time_us":14493,"prepare.run_wall_time_us":405672,"raft.queue_time_us":73391,"raft.run_cpu_time_us":30040,"raft.run_wall_time_us":69098,"replication_time_us":692197,"thread_start_us":226133,"threads_started":4,"wal-append.queue_time_us":100892}]]}
I20250831 07:58:46.303481 589 raft_consensus.cc:2953] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:46.328168 597 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:58:46.341229 646 log.cc:927] Time spent T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.152s user 0.001s sys 0.000s
I20250831 07:58:46.303058 519 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20250831 07:58:46.364790 519 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250831 07:58:46.579563 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 327850us
W20250831 07:58:46.716652 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 363878us
W20250831 07:58:46.839850 31775 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.009s sys 0.031s
W20250831 07:58:46.869242 31775 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s user 0.009s sys 0.031s
W20250831 07:58:46.868363 642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.011s sys 0.069s
W20250831 07:58:46.877058 642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.573s user 0.011s sys 0.070s
W20250831 07:58:46.839702 31804 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.012s sys 0.027s
I20250831 07:58:46.831435 535 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:46.895489 31804 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s user 0.012s sys 0.027s
I20250831 07:58:46.896430 578 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:46.862180 31873 thread.cc:641] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.009s sys 0.026s
I20250831 07:58:47.048511 31630 catalog_manager.cc:5582] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 2 to 3, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 3 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:47.049683 569 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:46.837965 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:47.052021 31628 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59664 (request call id 212) took 2090 ms. Trace:
I20250831 07:58:47.052498 31628 rpcz_store.cc:276] 0831 07:58:44.961562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:44.962719 (+ 1157us) service_pool.cc:225] Handling call
0831 07:58:47.051941 (+2089222us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":85928,"reported_tablets":3}
I20250831 07:58:47.050750 31630 catalog_manager.cc:5582] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 2, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
W20250831 07:58:47.042618 31873 thread.cc:608] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.753s user 0.009s sys 0.027s
W20250831 07:58:47.058552 541 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.026s sys 0.058s
W20250831 07:58:47.109989 541 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.762s user 0.026s sys 0.058s
I20250831 07:58:47.141609 658 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:47.142361 658 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:47.050226 569 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } attrs { replace: true } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:47.113992 599 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:58:47.148233 32436 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.836s user 0.064s sys 0.043s
I20250831 07:58:47.161702 496 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:47.198019 595 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:47.177848 500 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:58:47.189858 32436 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.886s user 0.064s sys 0.044s
I20250831 07:58:47.040970 535 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:47.208019 658 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:47.209316 500 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:47.204551 599 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:47.096027 578 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:47.198643 595 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:47.288405 653 log.cc:927] Time spent T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20250831 07:58:47.205561 496 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:47.229626 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 927763us
I20250831 07:58:47.214180 642 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:47.443849 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 271725us
W20250831 07:58:47.455417 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 303110us
I20250831 07:58:47.439329 568 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:47.462126 570 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:47.462883 629 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:47.443467 658 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:47.440263 467 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:47.441769 507 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:47.439690 540 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:47.610843 31802 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.017s sys 0.065s
W20250831 07:58:47.611402 31802 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.510s user 0.017s sys 0.065s
I20250831 07:58:47.469298 564 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:47.472512 602 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:47.631588 598 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:47.517155 535 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:47.647275 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:47.476608 642 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:47.649811 651 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:47.532994 570 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:47.632561 560 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:47.668274 496 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:47.616215 467 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:47.627437 507 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:47.618757 658 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:47.514729 568 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:47.687989 569 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:47.689456 568 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:47.653512 564 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:47.700516 664 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:47.651364 602 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:47.668900 535 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:47.680636 654 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:47.680583 560 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:47.677280 651 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:47.725636 570 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:47.701128 664 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:47.627808 540 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:47.759296 589 consensus_meta.cc:287] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.433s user 0.018s sys 0.002s
I20250831 07:58:47.704562 564 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:47.764618 595 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:47.765986 599 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:47.768715 496 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:47.769407 496 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:47.778000 668 log.cc:927] Time spent T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20250831 07:58:47.785228 602 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:47.714583 578 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:47.532750 629 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:47.666841 598 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:47.760191 663 log.cc:927] Time spent T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20250831 07:58:47.715206 535 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:47.771692 496 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:47.798028 540 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:47.801384 560 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:47.803043 660 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:47.804836 467 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:47.809718 598 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:47.794332 500 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:47.821283 660 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:47.779717 654 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:47.819485 496 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:47.806878 642 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:47.842077 642 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:47.843036 642 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:47.853382 629 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:47.838004 657 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:47.862339 657 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:47.866590 654 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:47.874097 642 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:47.881601 669 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:47.882454 669 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:47.990444 31729 raft_consensus.cc:2953] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:48.057066 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 33) took 1806 ms. Trace:
I20250831 07:58:48.057709 31804 rpcz_store.cc:276] 0831 07:58:46.249942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:46.255300 (+ 5358us) service_pool.cc:225] Handling call
0831 07:58:46.255573 (+ 273us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:46.292070 (+ 36497us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:46.292106 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:58:46.292569 (+ 463us) log.cc:844] Serialized 26 byte log entry
0831 07:58:46.900010 (+607441us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:58:46.900117 (+ 107us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:46.900223 (+ 106us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:58:48.056348 (+1156125us) raft_consensus.cc:1683] finished
0831 07:58:48.056393 (+ 45us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:48.056731 (+ 338us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7974,"raft.queue_time_us":26161,"spinlock_wait_cycles":8703360,"thread_start_us":635083,"threads_started":2,"wal-append.queue_time_us":615112}
I20250831 07:58:48.075394 496 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.054875 500 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.181536 560 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:48.182653 560 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.185825 507 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.187424 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 35) took 1353 ms. Trace:
I20250831 07:58:48.187866 31806 rpcz_store.cc:276] 0831 07:58:46.818618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:46.818733 (+ 115us) service_pool.cc:225] Handling call
0831 07:58:46.819034 (+ 301us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:46.819319 (+ 285us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:46.819346 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:58:46.819782 (+ 436us) log.cc:844] Serialized 26 byte log entry
0831 07:58:47.152970 (+333188us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:58:47.153023 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:47.153110 (+ 87us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:58:48.087154 (+934044us) raft_consensus.cc:1683] finished
0831 07:58:48.088936 (+ 1782us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:48.171674 (+ 82738us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":58742,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":167132,"thread_start_us":332740,"threads_started":1,"wal-append.queue_time_us":720694}
W20250831 07:58:48.190841 31805 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.342s user 0.010s sys 0.069s
I20250831 07:58:48.190994 598 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:58:48.191630 31805 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.361s user 0.010s sys 0.069s
I20250831 07:58:48.191994 598 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.199191 658 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.200484 535 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.201357 500 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.204658 651 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.208436 623 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.209647 623 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.216661 625 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.217816 625 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.211705 660 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.220377 500 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.220695 624 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.220743 575 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.195312 598 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:48.221786 624 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.223968 467 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.224836 467 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.225493 624 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.227469 467 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.219319 560 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.228068 496 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.221805 575 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.229735 672 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.229924 507 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.230053 467 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:48.254842 32433 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.012s sys 0.009s
I20250831 07:58:48.255407 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 919) took 1054 ms. Trace:
I20250831 07:58:48.261612 31727 rpcz_store.cc:276] 0831 07:58:47.200795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:47.200963 (+ 168us) service_pool.cc:225] Handling call
0831 07:58:48.253982 (+1053019us) spinlock_profiling.cc:243] Waited 3.2 ms on lock 0x7b5800000f98. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:58:48.254428 (+ 446us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:48.254775 (+ 347us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:48.254812 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:48.254890 (+ 78us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:58:48.254943 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:48.255004 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:48.255166 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7359360}
I20250831 07:58:48.238848 664 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:48.255424 32433 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.628s user 0.013s sys 0.009s
W20250831 07:58:48.272235 541 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.065s user 0.019s sys 0.055s
W20250831 07:58:48.272848 541 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.095s user 0.020s sys 0.056s
W20250831 07:58:48.273977 32436 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.010s sys 0.014s
W20250831 07:58:48.274406 32436 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s user 0.010s sys 0.014s
I20250831 07:58:48.275281 623 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.276680 541 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:48.276995 629 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.277908 629 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.285409 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 32) took 2035 ms. Trace:
I20250831 07:58:48.288512 624 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.263419 507 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.280126 625 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.277570 541 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.271692 669 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:48.299598 31730 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.805s user 0.034s sys 0.016s
I20250831 07:58:48.287910 31805 rpcz_store.cc:276] 0831 07:58:46.249670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:46.253014 (+ 3344us) service_pool.cc:225] Handling call
0831 07:58:46.253374 (+ 360us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:46.716059 (+462685us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:46.716089 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:58:46.721372 (+ 5283us) log.cc:844] Serialized 26 byte log entry
0831 07:58:48.192432 (+1471060us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:58:48.192537 (+ 105us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:48.192689 (+ 152us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:58:48.279424 (+ 86735us) raft_consensus.cc:1683] finished
0831 07:58:48.279454 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:48.285325 (+ 5871us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12414,"raft.queue_time_us":28199,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":8289,"thread_start_us":1627527,"threads_started":2,"wal-append.queue_time_us":1543821}
W20250831 07:58:48.301882 31775 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.824s user 0.011s sys 0.016s
W20250831 07:58:48.310309 31730 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.870s user 0.035s sys 0.017s
W20250831 07:58:48.306234 31734 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.742s user 0.007s sys 0.011s
I20250831 07:58:48.290279 568 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.289979 623 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:48.309386 31773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.021s sys 0.028s
W20250831 07:58:48.307994 32435 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.007s sys 0.053s
I20250831 07:58:48.300823 625 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:48.328097 31734 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.853s user 0.008s sys 0.012s
W20250831 07:58:48.314217 31775 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.842s user 0.012s sys 0.016s
W20250831 07:58:48.315968 31772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.013s sys 0.016s
W20250831 07:58:48.332104 32435 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.860s user 0.007s sys 0.054s
W20250831 07:58:48.335050 31772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.799s user 0.014s sys 0.016s
W20250831 07:58:48.329926 32434 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.014s sys 0.005s
W20250831 07:58:48.333117 31773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.827s user 0.022s sys 0.029s
W20250831 07:58:48.337391 32434 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.668s user 0.014s sys 0.005s
I20250831 07:58:48.305562 624 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.315897 679 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.323292 507 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:48.333003 31774 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.723s user 0.012s sys 0.006s
W20250831 07:58:48.336604 31700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.001s sys 0.018s
I20250831 07:58:48.315709 554 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:48.335435 31802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.015s sys 0.008s
I20250831 07:58:48.322712 560 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:48.312036 467 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.314026 677 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:48.336627 589 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.024s sys 0.012s
I20250831 07:58:48.327275 687 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:48.268915 672 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.349795 625 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:48.350895 31774 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.815s user 0.013s sys 0.007s
W20250831 07:58:48.351153 31700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.817s user 0.001s sys 0.019s
W20250831 07:58:48.351346 31802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.739s user 0.015s sys 0.008s
I20250831 07:58:48.321463 568 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.365334 672 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:48.336053 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 870128us
I20250831 07:58:48.352228 554 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.366577 560 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.368077 554 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.368920 560 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:48.348520 575 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.370441 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:48.371263 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:48.371662 31879 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:48.372210 31883 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:48.373140 568 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:48.353103 687 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.362046 679 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.375197 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:48.375959 31880 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:48.376295 679 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:48.358851 589 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s user 0.024s sys 0.014s
I20250831 07:58:48.377367 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:48.386178 31876 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:48.359069 623 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.366572 659 tablet_replica.cc:331] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10: stopping tablet replica
I20250831 07:58:48.395047 541 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:48.397537 613 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.405524 689 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.405640 613 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.406529 689 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.397797 669 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.408263 669 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.408669 629 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.410382 669 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.411005 690 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.398366 681 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.353633 677 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.398430 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:48.410295 625 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.412004 690 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.417548 693 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:48.414621 31801 raft_consensus.cc:2953] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:48.424419 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:48.425292 31874 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:48.425779 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:48.425801 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 61287.3us
I20250831 07:58:48.427294 31876 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
W20250831 07:58:48.427364 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 60961.3us
I20250831 07:58:48.429001 624 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:48.431524 692 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975: Append to log took a long time: real 0.054s user 0.001s sys 0.001s
I20250831 07:58:48.432395 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1378) took 1131 ms. Trace:
I20250831 07:58:48.438232 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1357) took 2108 ms. Trace:
I20250831 07:58:48.439038 688 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.439914 588 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.432807 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:48.418500 693 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.447244 588 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.448566 626 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:48.449574 589 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.457365 589 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.457751 623 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.458398 31698 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:48.458846 588 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:48.419418 687 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.441303 31802 rpcz_store.cc:276] 0831 07:58:46.329881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:46.330079 (+ 198us) service_pool.cc:225] Handling call
0831 07:58:46.717146 (+387067us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:46.717690 (+ 544us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:46.717731 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:58:46.718245 (+ 514us) log.cc:844] Serialized 26 byte log entry
0831 07:58:47.060962 (+342717us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:58:47.611879 (+550917us) log.cc:844] Serialized 14 byte log entry
0831 07:58:48.413404 (+801525us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:48.436189 (+ 22785us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:58:48.437961 (+ 1772us) raft_consensus.cc:1683] finished
0831 07:58:48.437975 (+ 14us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:48.438144 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":144545,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":176369,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":2913,"spinlock_wait_cycles":10089216,"thread_start_us":1532732,"threads_started":3,"wal-append.queue_time_us":662663}
I20250831 07:58:48.460887 689 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.464671 31698 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:58:48.440881 671 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.026s sys 0.059s
I20250831 07:58:48.455017 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 918) took 1260 ms. Trace:
I20250831 07:58:48.464694 31877 raft_consensus.cc:2466] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:48.464881 626 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.462811 560 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:48.468992 560 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.469406 560 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:48.440091 683 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.473098 626 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.432823 31734 rpcz_store.cc:276] 0831 07:58:47.300776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:47.448666 (+147890us) service_pool.cc:225] Handling call
0831 07:58:47.448978 (+ 312us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:47.449202 (+ 224us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 07:58:48.340316 (+891114us) log.cc:844] Serialized 14 byte log entry
0831 07:58:48.431917 (+ 91601us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:48.432012 (+ 95us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:58:48.432080 (+ 68us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:48.432172 (+ 92us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:48.432317 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":971790,"raft.queue_time_us":6699,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":915785,"threads_started":2}
I20250831 07:58:48.478161 31698 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:48.440007 681 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.463157 677 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.461480 589 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.444134 688 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.473449 683 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.465847 31730 rpcz_store.cc:276] 0831 07:58:47.194810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:47.195092 (+ 282us) service_pool.cc:225] Handling call
0831 07:58:47.195401 (+ 309us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:47.201149 (+ 5748us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 07:58:48.341661 (+1140512us) log.cc:844] Serialized 14 byte log entry
0831 07:58:48.454532 (+112871us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:48.454611 (+ 79us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:58:48.454701 (+ 90us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:48.454811 (+ 110us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:48.454967 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1119348,"raft.queue_time_us":131,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":136,"spinlock_wait_cycles":7415552,"thread_start_us":944534,"threads_started":2}
I20250831 07:58:48.633049 657 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.578307 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:48.558619 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:48.651852 570 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.560223 690 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.845026 570 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.846403 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:48.827219 560 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.847009 693 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.847285 31804 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:48.490898 569 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.581039 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:48.849332 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:48.606731 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:48.850257 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:58:48.490520 663 log.cc:927] Time spent T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.688s user 0.001s sys 0.014s
I20250831 07:58:48.606323 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:58:48.490485 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 62052us
I20250831 07:58:48.723806 674 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:48.608392 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:57332 (request call id 582) took 1775 ms. Trace:
I20250831 07:58:48.852006 674 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.852445 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:48.851708 32465 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:48.848249 31698 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:48.630741 467 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.526185 505 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:48.856366 505 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.856426 31698 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:48.856894 505 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:48.850663 31875 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:48.855553 679 raft_consensus.cc:2802] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:48.579522 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:48.842491 32468 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:48.852345 31881 rpcz_store.cc:276] 0831 07:58:46.832781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:46.832962 (+ 181us) service_pool.cc:225] Handling call
0831 07:58:48.575302 (+1742340us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b5400410854. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f81cd0 00007fdce8005eea 00007fdcecf5e60e 00007fdce65fbcc7 00007fdce65fbc52 00007fdce65fbb81 00007fdce65fbafc 00007fdce65fadb2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:48.608113 (+ 32811us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3661708544}
I20250831 07:58:48.612339 642 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.578696 561 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:48.852996 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:48.495146 671 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s user 0.027s sys 0.060s
I20250831 07:58:48.844463 683 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.858057 679 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.558979 613 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.636520 696 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.625245 587 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.559074 577 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:48.862171 507 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.848618 569 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.867792 594 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.868690 561 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.888263 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:48.887136 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:48.877635 679 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:48.885690 32469 raft_consensus.cc:1811] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Handling vote request from an unknown peer 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:48.892886 561 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:48.891637 613 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.904738 688 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:48.908473 570 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.925043 686 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:48.914703 577 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.925977 501 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:48.912490 674 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:48.892043 556 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.902199 507 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.927919 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:48.928414 31698 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:48.880293 642 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.923857 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:48.926967 560 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.941751 32467 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:48.939455 31808 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:48.936120 501 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.936460 613 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.909929 500 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:48.929714 686 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.945660 686 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:48.947284 500 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.949343 500 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.915895 681 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.902365 696 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.914104 594 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.911152 587 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.955905 557 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:48.956629 557 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:48.958487 594 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.951996 32469 raft_consensus.cc:2408] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250831 07:58:48.930053 598 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:48.959194 600 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:48.974920 569 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:48.974825 31847 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:48.974643 600 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.975642 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:48.973358 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 52733.9us
I20250831 07:58:48.978058 31698 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:48.952479 672 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:48.958491 557 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:48.944701 655 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:48.978590 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:48.983949 696 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:48.976961 683 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:48.985251 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:48.965981 613 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.000164 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:48.998622 31698 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:48.989245 624 raft_consensus.cc:2802] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:49.002130 31847 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:48.989058 600 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:48.993566 560 raft_consensus.cc:2802] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:49.014853 500 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:49.014889 597 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:49.004403 594 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:48.964519 556 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.993436 683 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:48.994211 672 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.034036 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.031469 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:49.031726 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.013779 624 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:48.999688 693 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:49.033216 504 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:49.045701 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.034664 31731 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:49.045997 504 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.018232 500 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:49.047281 500 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:49.048208 504 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.018620 560 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:49.024412 674 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:49.052778 674 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.054347 693 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.000298 598 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.043921 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:49.035802 31878 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:49.026371 594 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:49.089462 594 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:49.090083 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.025018 667 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:49.021683 575 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:49.141417 575 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:49.141882 575 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:49.052188 560 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:49.040786 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:49.029959 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:49.040817 544 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:49.084725 627 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:49.085356 504 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:49.085043 597 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:49.085958 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.020728 655 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.030532 577 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.155696 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.156225 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:49.140856 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:49.157183 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.157752 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.146256 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.148566 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.147677 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 45) took 1531 ms. Trace:
W20250831 07:58:49.139402 698 log.cc:927] Time spent T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.224s user 0.000s sys 0.001s
I20250831 07:58:49.153916 504 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.154672 627 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.155423 655 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.156352 31733 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:49.164338 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.157774 31876 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:49.043403 624 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:49.159183 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.149744 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.158900 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.162190 549 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:49.149055 688 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:49.164489 667 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:49.174682 688 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.176401 667 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:49.162710 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.176805 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.177038 688 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.163372 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.163779 501 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:49.165099 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.174633 549 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.178249 31878 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:58:49.166733 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.147240 626 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:49.179942 549 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.148476 544 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.169453 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.170727 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.147831 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.168990 597 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:49.164497 577 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:49.040926 32471 raft_consensus.cc:2953] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:49.179869 626 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.186035 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.159109 31882 rpcz_store.cc:276] 0831 07:58:47.445589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:47.446006 (+ 417us) service_pool.cc:225] Handling call
0831 07:58:48.462413 (+1016407us) spinlock_profiling.cc:243] Waited 3.2 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:58:48.462695 (+ 282us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:48.463041 (+ 346us) raft_consensus.cc:1532] Early marking committed up to index 5
0831 07:58:48.866792 (+403751us) log.cc:844] Serialized 14 byte log entry
0831 07:58:48.969437 (+102645us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:48.969499 (+ 62us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:58:48.969553 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:48.969654 (+ 101us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:48.976419 (+ 6765us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":9655,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":55,"raft.queue_time_us":110,"raft.run_cpu_time_us":381,"raft.run_wall_time_us":9582,"spinlock_wait_cycles":7364992,"thread_start_us":483241,"threads_started":2,"wal-append.queue_time_us":136635}
I20250831 07:58:49.227841 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.228631 31802 raft_consensus.cc:1811] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Handling vote request from an unknown peer 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:49.229357 31802 raft_consensus.cc:2408] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250831 07:58:49.231142 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.193032 577 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:49.233119 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.233263 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.232425 507 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:49.233692 672 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.258140 31807 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:49.251493 32435 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:49.184922 642 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.298979 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.233937 693 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.195534 683 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:58:49.311431 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 61597.6us
I20250831 07:58:49.314971 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.315114 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.316368 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.317063 31733 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:49.318980 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.320060 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.320986 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.299598 587 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:49.327116 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.328814 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.330556 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.332248 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.333374 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.333573 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.332295 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.335901 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.336064 571 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:49.330714 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.336640 571 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.337123 31809 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:49.338809 571 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:49.339670 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.340116 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.340821 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.341442 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.341751 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.342942 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.344143 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.306742 598 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.304258 674 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:49.349517 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 183670us
I20250831 07:58:49.321002 642 raft_consensus.cc:2802] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:49.351753 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.355005 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.356211 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.335642 467 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:49.349037 504 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.358012 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.368153 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.360347 659 raft_consensus.cc:2241] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 07:58:49.355055 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.362545 561 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:49.363420 594 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:49.364343 686 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:49.350577 702 log.cc:927] Time spent T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.204s user 0.001s sys 0.000s
I20250831 07:58:49.356539 31802 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:49.368791 31733 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
W20250831 07:58:49.370959 505 consensus_meta.cc:287] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.513s user 0.009s sys 0.010s
I20250831 07:58:49.355315 642 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:49.371433 556 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:49.372208 544 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.372390 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.372655 31731 raft_consensus.cc:1238] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:49.366820 627 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.379518 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.383702 31698 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:49.386549 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.393448 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.401091 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.401687 626 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.402379 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.402551 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.403008 577 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:49.401190 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.405825 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.326315 560 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.414911 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.421631 31730 raft_consensus.cc:1238] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:49.421875 31698 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:58:49.422179 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 167820us
I20250831 07:58:49.425974 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.427636 31878 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:49.362576 500 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.412083 642 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:49.426167 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.432279 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:49.439216 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 60023.9us
I20250831 07:58:49.440441 32435 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:49.440491 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.424662 544 raft_consensus.cc:2802] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:49.441336 31878 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:49.425642 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.448707 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.432428 556 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:58:49.455003 556 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:49.455294 659 raft_consensus.cc:2270] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 07:58:49.455482 556 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:49.456817 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.454952 31732 raft_consensus.cc:1238] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:49.448801 32436 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:49.453275 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.457711 31728 raft_consensus.cc:1238] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:49.461215 686 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.464368 31698 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:49.469499 659 ts_tablet_manager.cc:1905] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:58:49.469844 575 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:49.471176 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:49.472497 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:49.473908 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:49.477147 714 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:49.471208 31883 raft_consensus.cc:1238] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:49.477602 714 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:49.477964 714 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:49.421742 467 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:49.481531 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 47355us
I20250831 07:58:49.483664 32436 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:49.451181 544 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:49.412361 505 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.495967 31880 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:49.496954 31698 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:49.498333 31732 raft_consensus.cc:1238] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:49.478632 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.498528 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:49.500028 672 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:49.500772 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.501122 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.502192 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.503505 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.503731 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.504176 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.504696 500 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.482857 686 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:49.505450 686 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:49.505818 686 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:49.506139 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.506505 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.507107 31733 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:49.504377 31727 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:49.489714 544 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:49.504143 31731 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:49.519845 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.520833 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:49.521131 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:49.521013 31627 catalog_manager.cc:5582] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 2, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 2 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:49.521777 31874 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:49.522838 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:49.523365 31878 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:49.529116 575 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:49.529457 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.530181 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.530762 31730 raft_consensus.cc:2466] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:49.520862 31630 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59666 (request call id 234) took 4110 ms. Trace:
I20250831 07:58:49.491024 560 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.544387 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.545512 31730 raft_consensus.cc:2391] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:49.545799 31733 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:49.546198 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:49.535283 32436 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:49.546900 31875 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:49.548491 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:49.549170 31879 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:49.550704 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.552192 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.553457 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.554203 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.556665 716 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:49.547420 690 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:49.557804 690 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:49.558262 690 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:49.554167 31732 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:49.510298 561 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:49.500614 672 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:49.517072 598 raft_consensus.cc:2802] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:49.583034 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:49.583465 712 log.cc:927] Time spent T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20250831 07:58:49.576417 594 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:49.532021 31630 rpcz_store.cc:276] 0831 07:58:45.390812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:45.390973 (+ 161us) service_pool.cc:225] Handling call
0831 07:58:49.500667 (+4109694us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:58:47.637794 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:58:47.673538 (+ 35744us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:58:47.673579 (+ 41us) write_op.cc:435] Acquired schema lock
0831 07:58:47.673598 (+ 19us) tablet.cc:660] Decoding operations
0831 07:58:47.674627 (+ 1029us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:58:47.674699 (+ 72us) write_op.cc:641] Partition lock acquired for write op
0831 07:58:47.674736 (+ 37us) tablet.cc:683] Acquiring locks for 2 operations
0831 07:58:47.675104 (+ 368us) tablet.cc:699] Row locks acquired
0831 07:58:47.675126 (+ 22us) write_op.cc:260] PREPARE: finished
0831 07:58:47.675293 (+ 167us) write_op.cc:270] Start()
0831 07:58:47.675424 (+ 131us) write_op.cc:276] Timestamp: P: 1756627127675238 usec, L: 0
0831 07:58:47.675451 (+ 27us) op_driver.cc:348] REPLICATION: starting
0831 07:58:47.724770 (+ 49319us) log.cc:844] Serialized 923 byte log entry
0831 07:58:48.980065 (+1255295us) op_driver.cc:464] REPLICATION: finished
0831 07:58:49.007394 (+ 27329us) spinlock_profiling.cc:243] Waited 6.27 ms on lock 0x7b1800242b00. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b8bf 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:58:49.324602 (+317208us) write_op.cc:301] APPLY: starting
0831 07:58:49.346336 (+ 21734us) tablet.cc:1365] starting BulkCheckPresence
0831 07:58:49.346532 (+ 196us) tablet.cc:1368] finished BulkCheckPresence
0831 07:58:49.346572 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:58:49.347081 (+ 509us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:58:49.348523 (+ 1442us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0831 07:58:49.348610 (+ 87us) write_op.cc:312] APPLY: finished
0831 07:58:49.371590 (+ 22980us) log.cc:844] Serialized 29 byte log entry
0831 07:58:49.371781 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:49.371924 (+ 143us) write_op.cc:454] Released schema lock
0831 07:58:49.371988 (+ 64us) write_op.cc:341] FINISH: Updating metrics
0831 07:58:49.372439 (+ 451us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:49.372467 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":706119,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":318968,"apply.run_cpu_time_us":2657,"apply.run_wall_time_us":47951,"num_ops":2,"prepare.queue_time_us":370708,"prepare.run_cpu_time_us":103196,"prepare.run_wall_time_us":1375973,"raft.queue_time_us":483553,"raft.run_cpu_time_us":27924,"raft.run_wall_time_us":273051,"replication_time_us":1266062,"spinlock_wait_cycles":14429952,"thread_start_us":1722444,"threads_started":4,"wal-append.queue_time_us":615470}]]}
W20250831 07:58:49.608554 710 log.cc:927] Time spent T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20250831 07:58:49.609722 713 log.cc:927] Time spent T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250831 07:58:49.610136 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.610812 31730 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:49.616474 31698 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:49.616451 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.619030 31733 raft_consensus.cc:2391] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
W20250831 07:58:49.621771 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 103207us
I20250831 07:58:49.622644 32436 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:49.624387 31698 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:49.626194 31698 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:49.627149 31698 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:49.573258 598 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:49.629009 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.530650 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.628145 31773 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:49.633221 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:49.631675 31730 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:49.630980 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.616146 677 raft_consensus.cc:2747] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:49.636438 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.636464 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.638363 31882 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:49.647977 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:49.628340 561 raft_consensus.cc:2747] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:49.624790 31874 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:58:49.629146 597 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:49.634085 598 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:49.660701 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.631407 501 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:49.683625 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.684857 31773 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:49.684433 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:49.676872 31879 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:58:49.689492 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.696148 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.694936 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.702347 31731 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:49.702790 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.703421 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.704047 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.696120 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.704974 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:49.705986 719 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:49.706458 719 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:49.704594 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.706941 719 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:49.707474 31731 raft_consensus.cc:2391] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:49.689972 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:49.699167 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:49.704136 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:49.701723 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.696540 31732 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:49.701573 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.705255 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.716418 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.716850 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.654716 626 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:49.704278 686 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:49.636358 557 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:58:49.723723 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 59748.4us
I20250831 07:58:49.732209 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.735774 31807 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:49.738066 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.738973 31883 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:49.741979 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.739449 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.744307 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.746032 31807 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:49.741598 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:49.749629 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.750720 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.751761 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.752413 31730 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:49.754750 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.747404 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.757516 31875 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:58:49.758750 31728 raft_consensus.cc:1238] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:49.727074 626 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:49.730360 711 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:49.760187 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.670607 667 consensus_queue.cc:1046] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
I20250831 07:58:49.762346 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:49.732355 679 consensus_meta.cc:287] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.825s user 0.017s sys 0.005s
I20250831 07:58:49.739807 505 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.730635 31803 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:50.013089 32436 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.118148 711 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:50.118422 613 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:50.118855 711 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:49.980832 693 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:50.118996 613 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:58:50.119450 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 106469us
I20250831 07:58:50.119506 613 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.110675 560 raft_consensus.cc:3053] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:49.946834 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:50.055088 690 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:49.769024 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.824858 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:49.892760 642 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.122517 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:50.122763 690 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:50.090842 575 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:50.035949 31849 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.123661 575 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.008498 31848 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:50.124351 32463 raft_consensus.cc:1238] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
W20250831 07:58:50.124675 31848 raft_consensus.cc:2684] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250831 07:58:49.865192 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:50.125211 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 116795us
I20250831 07:58:49.823346 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:49.991695 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:50.125618 575 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:50.129125 629 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:50.147796 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:49.946034 31878 raft_consensus.cc:2466] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
W20250831 07:58:49.959568 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 181415us
W20250831 07:58:50.059362 556 consensus_meta.cc:287] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.603s user 0.007s sys 0.002s
W20250831 07:58:49.957057 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 304093us
I20250831 07:58:49.873737 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:49.980924 624 consensus_meta.cc:287] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.791s user 0.006s sys 0.005s
I20250831 07:58:50.124378 642 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:50.151553 575 consensus_queue.cc:1046] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:58:50.151890 624 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.152580 571 raft_consensus.cc:3053] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:50.152957 571 raft_consensus.cc:738] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:50.150223 32434 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.154026 624 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:50.153810 571 consensus_queue.cc:260] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:49.958151 467 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:49.765890 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:50.110905 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:50.155874 31732 raft_consensus.cc:2391] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:50.156131 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:50.150068 556 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.156894 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:58:50.154958 32436 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.509s (SENT) [suppressed 1 similar messages]
I20250831 07:58:50.131202 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:50.158768 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:50.155254 31881 raft_consensus.cc:1238] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:50.159615 31726 raft_consensus.cc:1238] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:58:50.071156 544 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.160552 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1286) took 1009 ms. Trace:
I20250831 07:58:50.160881 31881 rpcz_store.cc:276] 0831 07:58:49.152907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:49.168672 (+ 15765us) service_pool.cc:225] Handling call
0831 07:58:49.168939 (+ 267us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:50.159507 (+990568us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:50.160356 (+ 849us) spinlock_profiling.cc:243] Waited 986 ms on lock 0x7b54002bd254. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:50.160500 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2267971968}
I20250831 07:58:49.766316 626 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.162267 544 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:49.765123 672 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:50.162917 31727 raft_consensus.cc:1238] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:50.103381 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:50.106042 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:50.125850 31848 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:58:49.827106 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 83171us
W20250831 07:58:50.137504 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 113423us
I20250831 07:58:50.035300 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:50.155007 571 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.167255 32471 raft_consensus.cc:1238] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:50.168754 667 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:58:50.058020 714 consensus_meta.cc:287] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.580s user 0.007s sys 0.008s
I20250831 07:58:50.169272 667 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.169637 714 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.170728 31848 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:50.154054 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:50.172614 31849 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.154887 32434 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.152041 560 raft_consensus.cc:738] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:50.004944 686 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:50.176280 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:50.178064 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:50.178597 31880 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:49.865794 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:50.179893 31698 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.183055 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:49.866467 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:50.187958 575 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:50.188840 568 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.190246 725 consensus_queue.cc:1046] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:58:50.190550 31848 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:49.969138 598 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.191011 714 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:50.191916 561 raft_consensus.cc:2747] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:58:50.192267 556 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:50.066995 659 ts_tablet_manager.cc:1918] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:58:50.172573 467 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:50.192835 32465 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:50.191885 31698 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.193414 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:50.194115 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:50.194443 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:50.195578 667 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:50.196445 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:49.911898 703 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:50.212433 598 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:50.212600 703 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.214649 703 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:50.214717 725 raft_consensus.cc:3053] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:50.170692 642 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.199569 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:50.206630 575 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:50.187168 690 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.083449 719 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.197378 693 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.170804 679 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.215811 575 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:50.221976 679 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:50.222226 575 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:50.216248 667 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:50.223116 667 raft_consensus.cc:695] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:50.224095 667 consensus_queue.cc:237] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.216341 31698 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.170817 31732 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e"
dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:48724
I20250831 07:58:50.204821 560 consensus_queue.cc:260] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.227524 31732 raft_consensus.cc:604] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:50.228017 560 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.204890 31731 raft_consensus.cc:2391] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:50.229705 659 log.cc:1199] T d3d650c422ee41af95705f3fd876ddb8 P 8247a38b550649aa9cd6e7668dcaec10: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-2-root/wals/d3d650c422ee41af95705f3fd876ddb8
I20250831 07:58:50.219414 693 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:50.231156 659 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57332 (request call id 580) took 4064 ms. Trace:
I20250831 07:58:50.209956 467 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:50.232371 467 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.220697 577 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:50.202705 31730 raft_consensus.cc:1238] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:50.211444 31729 raft_consensus.cc:1238] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:50.233276 577 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.220947 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:50.234936 31698 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.235355 577 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:50.235832 31698 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.215154 725 raft_consensus.cc:738] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:50.130843 613 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.231563 693 consensus_queue.cc:1046] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:50.220455 674 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:50.206812 31849 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.221328 31725 raft_consensus.cc:1238] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:50.247345 31806 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:37044
I20250831 07:58:50.248008 31806 raft_consensus.cc:604] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:50.248319 31806 raft_consensus.cc:612] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:58:50.232324 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:50.207775 32434 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.217934 31880 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:50.220258 31773 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:58:50.236865 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 81932.5us
I20250831 07:58:50.236001 594 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:50.219776 629 raft_consensus.cc:2747] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:50.222013 688 consensus_queue.cc:1046] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20250831 07:58:50.244738 703 raft_consensus.cc:2802] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:50.253371 32436 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:50.254297 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 47579.7us
I20250831 07:58:50.256006 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:50.256161 725 consensus_queue.cc:260] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.257138 725 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.218017 719 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:50.243242 597 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:50.261284 32436 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:50.262759 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 55101.6us
I20250831 07:58:50.263270 32436 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.264707 32436 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.264870 32434 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.266376 717 raft_consensus.cc:2747] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.267166 613 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:50.267270 714 raft_consensus.cc:2747] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.267952 716 raft_consensus.cc:2747] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.268239 32465 raft_consensus.cc:1238] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:50.267568 577 raft_consensus.cc:3053] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:50.268659 577 raft_consensus.cc:738] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:50.221572 656 consensus_queue.cc:1046] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20250831 07:58:50.262706 690 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.261097 674 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.268646 677 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.267200 31849 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.221129 683 consensus_queue.cc:1046] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20250831 07:58:50.274343 31698 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.260473 571 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:50.270661 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 62751us
I20250831 07:58:50.271608 594 raft_consensus.cc:695] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:50.281522 597 raft_consensus.cc:695] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:50.271230 577 consensus_queue.cc:260] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.271445 642 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.282721 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:50.289271 31880 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:50.262169 703 raft_consensus.cc:695] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:58:50.278003 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 55089.3us
I20250831 07:58:50.287798 597 consensus_queue.cc:237] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:50.239472 686 consensus_queue.cc:1162] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7ecb257174554a26b2c35ad04b0eabae" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20250831 07:58:50.295476 598 consensus_queue.cc:1046] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:58:50.297237 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:50.297708 598 raft_consensus.cc:3053] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:50.298100 598 raft_consensus.cc:738] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:50.298400 674 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:50.298658 32468 raft_consensus.cc:1238] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:50.289736 629 raft_consensus.cc:2747] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.290956 31775 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.291524 31698 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.291893 31849 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:50.292559 31806 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:37044
I20250831 07:58:50.290057 679 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:50.295104 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:50.295045 467 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.290515 31848 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:50.261282 561 consensus_queue.cc:1046] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20250831 07:58:50.307132 31883 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:50.308458 31848 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:50.310761 31775 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.301641 31806 raft_consensus.cc:604] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:50.298938 674 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.268456 711 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.277112 626 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.306597 577 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:50.289774 681 consensus_queue.cc:1046] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20250831 07:58:50.312837 500 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:58:50.346899 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:50.315969 568 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:58:50.353204 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 58806.1us
W20250831 07:58:50.359097 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 48432.3us
W20250831 07:58:50.348060 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 77759.2us
I20250831 07:58:50.366531 31880 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:58:50.367224 31775 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.298913 598 consensus_queue.cc:260] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.371634 31880 raft_consensus.cc:1238] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:50.325376 679 raft_consensus.cc:695] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:50.287669 594 consensus_queue.cc:237] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:50.373879 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:50.375963 31880 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.377589 31698 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:58:50.364972 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 100197us
I20250831 07:58:50.382478 32434 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:50.381435 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:50.381836 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:50.370481 731 raft_consensus.cc:3053] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:50.386274 31698 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.378545 688 raft_consensus.cc:2747] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:58:50.393978 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 85603.5us
I20250831 07:58:50.315562 703 consensus_queue.cc:237] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.370826 624 consensus_queue.cc:1046] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20250831 07:58:50.389899 731 raft_consensus.cc:738] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:50.394402 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:50.390321 735 raft_consensus.cc:2747] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.231662 659 rpcz_store.cc:276] 0831 07:58:46.166598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:46.167278 (+ 680us) service_pool.cc:225] Handling call
0831 07:58:46.800999 (+633721us) ts_tablet_manager.cc:1145] Deleting tablet d3d650c422ee41af95705f3fd876ddb8
0831 07:58:46.801090 (+ 91us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0831 07:58:48.366044 (+1564954us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b5400057654. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce8005fa9 00007fdcecfaa282 00007fdcecfda260 00007fdcecfc163d 00007fdcecfc15e9 00007fdcecfc1579 00007fdcecfc1541 00007fdcecfc076d 0000000000379254 0000000000379099
0831 07:58:49.856782 (+1490738us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:50.066669 (+209887us) tablet_metadata.cc:641] Metadata flushed
0831 07:58:50.231062 (+164393us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":961461,"spinlock_wait_cycles":867938560,"tablet-delete.queue_time_us":518194,"thread_start_us":813422,"threads_started":1}
I20250831 07:58:50.382896 629 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:50.397221 31806 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:37044
I20250831 07:58:50.386437 560 raft_consensus.cc:2802] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:50.383491 568 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250831 07:58:50.388181 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 96400.7us
I20250831 07:58:50.465350 31699 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:50.480858 31880 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:50.468253 31775 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.381512 730 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:58:50.468946 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 68104us
I20250831 07:58:50.401932 667 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.480675 31806 raft_consensus.cc:604] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:50.387140 500 consensus_queue.cc:1046] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20250831 07:58:50.480998 31883 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
W20250831 07:58:50.464953 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 65373.5us
I20250831 07:58:50.493058 31849 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.387712 642 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250831 07:58:50.497655 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 217878us
I20250831 07:58:50.494908 31698 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.498095 31775 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.498545 556 raft_consensus.cc:2747] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.498839 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:50.498988 711 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:50.476459 31874 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.469811 575 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.500523 736 raft_consensus.cc:2747] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.500564 738 raft_consensus.cc:2747] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.502291 679 consensus_queue.cc:237] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.480643 598 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.482612 629 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:50.514215 467 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:50.515127 31699 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.516381 735 raft_consensus.cc:2802] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:50.518118 575 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:50.516526 31775 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.503572 560 raft_consensus.cc:695] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:50.508930 642 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:50.492728 568 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:50.526695 732 log.cc:927] Time spent T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20250831 07:58:50.527554 31775 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.524282 626 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:50.546555 672 raft_consensus.cc:3053] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:50.530665 624 raft_consensus.cc:3053] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:50.546936 704 raft_consensus.cc:2747] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.547252 672 raft_consensus.cc:738] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:50.547394 624 raft_consensus.cc:738] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:50.548355 672 consensus_queue.cc:260] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.548480 624 consensus_queue.cc:260] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.549521 672 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.549753 624 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.523593 629 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:50.553339 31775 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.554772 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:50.530086 500 raft_consensus.cc:2802] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:50.558125 31699 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.525573 735 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:50.531152 642 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.531539 693 raft_consensus.cc:2802] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:50.561415 31802 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
mode: GRACEFUL
new_leader_uuid: "8247a38b550649aa9cd6e7668dcaec10"
from {username='slave'} at 127.0.0.1:37044
I20250831 07:58:50.561873 31699 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.544739 568 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.562904 31802 raft_consensus.cc:604] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Received request to transfer leadership to TS 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.546597 560 consensus_queue.cc:237] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.563953 667 raft_consensus.cc:3053] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:50.564352 667 raft_consensus.cc:738] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:50.565203 667 consensus_queue.cc:260] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.546597 744 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.566196 31698 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.566392 667 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.571522 31775 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.572918 31775 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.574188 31775 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.581112 31699 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:50.582916 31699 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.584648 31699 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.587002 31883 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:58:50.589951 31883 raft_consensus.cc:604] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:50.590394 31883 raft_consensus.cc:612] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:58:50.586140 561 raft_consensus.cc:2747] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:50.591897 544 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:50.592859 31881 raft_consensus.cc:1238] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:50.585608 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:50.594676 31775 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.592409 544 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:50.538728 731 consensus_queue.cc:260] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.588431 711 raft_consensus.cc:2747] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.617192 711 raft_consensus.cc:991] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65: : Instructing follower 7ecb257174554a26b2c35ad04b0eabae to start an election
W20250831 07:58:50.617487 692 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975: Append to log took a long time: real 0.116s user 0.003s sys 0.000s
I20250831 07:58:50.617628 711 raft_consensus.cc:1079] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Signalling peer 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:50.606135 674 raft_consensus.cc:3053] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:50.619210 674 raft_consensus.cc:738] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:50.597724 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250831 07:58:50.620035 32465 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60"
dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:38362
I20250831 07:58:50.620703 32465 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:50.621043 32465 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.617399 704 raft_consensus.cc:2802] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:50.575706 742 raft_consensus.cc:2747] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.568467 594 raft_consensus.cc:2747] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.561432 735 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.635579 571 raft_consensus.cc:2747] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.638494 686 raft_consensus.cc:991] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c: : Instructing follower 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:50.639153 686 raft_consensus.cc:1079] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Signalling peer 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:50.642027 32466 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e"
dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:38366
I20250831 07:58:50.642179 629 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.642796 32466 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:50.643353 32466 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:50.644484 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 49920.8us
I20250831 07:58:50.645156 31775 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.557039 743 raft_consensus.cc:2802] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:50.606272 613 raft_consensus.cc:3053] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:50.588291 725 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:58:50.583649 693 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:50.593114 626 raft_consensus.cc:2747] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250831 07:58:50.653385 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 68829.5us
I20250831 07:58:50.690739 31699 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.592581 556 raft_consensus.cc:2747] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:58:50.654304 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 161314us
I20250831 07:58:50.592612 736 raft_consensus.cc:2802] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:50.621747 704 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:50.604243 688 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250831 07:58:50.665849 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 153546us
I20250831 07:58:50.593044 500 raft_consensus.cc:695] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:58:50.692543 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 47469.2us
I20250831 07:58:50.622093 544 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:50.705957 31775 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.692188 731 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.699508 31699 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:50.664052 743 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:50.708423 31775 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.697194 31849 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.688110 613 raft_consensus.cc:738] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:50.694489 693 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.716216 31699 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.701575 577 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:50.715502 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:58:50.715902 627 raft_consensus.cc:2747] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:50.700099 736 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:50.732421 31849 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.736359 467 raft_consensus.cc:2747] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.736735 31849 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.737852 31849 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.718535 629 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:50.719431 31775 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.740520 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:50.602308 738 consensus_queue.cc:1046] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.100s
I20250831 07:58:50.720294 32434 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:50.738742 31874 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:50.741272 747 raft_consensus.cc:2747] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.703727 704 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.730948 505 raft_consensus.cc:2747] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.722014 740 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.723203 500 consensus_queue.cc:237] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:50.723860 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:50.724274 31616 catalog_manager.cc:4928] TS 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): tablet d3d650c422ee41af95705f3fd876ddb8 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
I20250831 07:58:50.702797 688 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:50.700457 752 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:50.724982 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:50.719900 743 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.620057 674 consensus_queue.cc:260] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.714046 742 raft_consensus.cc:2802] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:50.726255 568 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.757433 32466 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.730093 31699 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:50.741926 656 raft_consensus.cc:3053] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:50.706467 725 raft_consensus.cc:695] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:50.732354 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:50.732064 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:50.732813 31627 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59674 (request call id 213) took 4917 ms. Trace:
I20250831 07:58:50.739096 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:50.740237 31849 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.743435 31698 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:50.744304 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:50.746304 711 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:50.768174 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:50.746559 738 raft_consensus.cc:3053] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:50.730777 577 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:50.756029 613 consensus_queue.cc:260] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.744232 32465 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:50.727082 626 raft_consensus.cc:2747] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:50.737299 736 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:50.740828 744 raft_consensus.cc:2747] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.704689 686 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:50.766603 568 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:50.767372 32466 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:50.749768 688 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.864820 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:50.770332 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:50.914700 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:50.865286 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:51.047492 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 307288us
I20250831 07:58:51.051944 31849 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.053071 31849 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.913417 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:50.773029 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:50.773015 31881 raft_consensus.cc:1238] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:50.865494 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:50.938823 32471 raft_consensus.cc:1238] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:50.852978 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:50.973681 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:58:50.927860 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 155319us
I20250831 07:58:50.926571 31848 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:51.056289 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:50.852438 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:50.852052 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:50.852265 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:50.974444 31734 raft_consensus.cc:1238] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:50.969717 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:50.972692 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:51.049480 31725 raft_consensus.cc:2391] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:50.907785 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:50.850553 31628 catalog_manager.cc:5582] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 2, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 2 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:51.088675 31875 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:51.057559 31849 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:51.068316 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 141833us
I20250831 07:58:51.058452 31876 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:51.098098 31849 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.761832 742 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:51.047101 577 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.100172 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:50.973022 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:51.101187 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 357801us
I20250831 07:58:50.974062 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:50.973711 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:50.763954 674 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:51.125814 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:50.887444 749 log.cc:927] Time spent T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.238s user 0.002s sys 0.000s
I20250831 07:58:50.881937 734 raft_consensus.cc:991] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65: : Instructing follower 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:50.768685 656 raft_consensus.cc:738] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:51.107656 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:50.764336 505 raft_consensus.cc:2747] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:51.108001 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:51.093712 747 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.108665 31628 catalog_manager.cc:5582] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 2, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 2 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:51.111740 31849 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.018035 31627 rpcz_store.cc:276] 0831 07:58:45.815492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:45.815683 (+ 191us) service_pool.cc:225] Handling call
0831 07:58:45.934466 (+118783us) spinlock_profiling.cc:243] Waited 1.92 ms on lock 0x7b54000af354. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f81cd0 00007fdce8005eea 00007fdcedd2417c 00007fdcede1cb5e 00007fdcec365484 00007fdcec365412 00007fdcec365341 00007fdcec3652bc 00007fdcec364572 00007fdce63fcccc 00007fdce63fc106
0831 07:58:50.732727 (+4798261us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:58:50.020058 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:58:50.020773 (+ 715us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:58:50.020835 (+ 62us) write_op.cc:435] Acquired schema lock
0831 07:58:50.020860 (+ 25us) tablet.cc:660] Decoding operations
0831 07:58:50.021313 (+ 453us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:58:50.021410 (+ 97us) write_op.cc:641] Partition lock acquired for write op
0831 07:58:50.021434 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0831 07:58:50.021755 (+ 321us) tablet.cc:699] Row locks acquired
0831 07:58:50.021780 (+ 25us) write_op.cc:260] PREPARE: finished
0831 07:58:50.021979 (+ 199us) write_op.cc:270] Start()
0831 07:58:50.022179 (+ 200us) write_op.cc:276] Timestamp: P: 1756627130021907 usec, L: 0
0831 07:58:50.022201 (+ 22us) op_driver.cc:348] REPLICATION: starting
0831 07:58:50.023003 (+ 802us) log.cc:844] Serialized 538 byte log entry
0831 07:58:50.088325 (+ 65322us) op_driver.cc:464] REPLICATION: finished
0831 07:58:50.471620 (+383295us) write_op.cc:301] APPLY: starting
0831 07:58:50.480384 (+ 8764us) tablet.cc:1365] starting BulkCheckPresence
0831 07:58:50.497091 (+ 16707us) tablet.cc:1368] finished BulkCheckPresence
0831 07:58:50.500697 (+ 3606us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:58:50.500937 (+ 240us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:58:50.501106 (+ 169us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0831 07:58:50.501148 (+ 42us) write_op.cc:312] APPLY: finished
0831 07:58:50.501337 (+ 189us) log.cc:844] Serialized 23 byte log entry
0831 07:58:50.501494 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:50.501570 (+ 76us) write_op.cc:454] Released schema lock
0831 07:58:50.501607 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0831 07:58:50.501929 (+ 322us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:50.501955 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"spinlock_wait_cycles":4404992,"child_traces":[["op",{"apply.queue_time_us":311198,"apply.run_cpu_time_us":1683,"apply.run_wall_time_us":35273,"num_ops":1,"prepare.queue_time_us":389676,"prepare.run_cpu_time_us":3316,"prepare.run_wall_time_us":14980,"replication_time_us":65935,"thread_start_us":275905,"threads_started":2}]]}
I20250831 07:58:51.112833 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:50.889585 725 consensus_queue.cc:237] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.114481 714 consensus_queue.cc:1046] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250831 07:58:51.018011 738 raft_consensus.cc:738] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:51.116429 31848 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:51.116878 624 raft_consensus.cc:2747] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.115561 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:51.120365 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:50.763675 683 consensus_queue.cc:1046] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.337s
I20250831 07:58:51.121562 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:51.061115 739 raft_consensus.cc:2747] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.943480 731 consensus_queue.cc:1046] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.167s
I20250831 07:58:51.101231 568 raft_consensus.cc:2747] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:58:51.016803 686 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:51.056605 745 raft_consensus.cc:2747] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:50.939953 704 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.100337 677 raft_consensus.cc:2762] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250831 07:58:51.136276 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:51.103907 751 raft_consensus.cc:2747] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:51.141378 31727 raft_consensus.cc:3053] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:51.141599 31849 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:50.943254 693 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.889680 642 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.151476 31628 catalog_manager.cc:5582] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 2, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 2 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:51.190724 31628 catalog_manager.cc:5582] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:50.946861 735 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:50.762781 753 consensus_queue.cc:1046] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.372s
I20250831 07:58:51.107904 742 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:50.887079 544 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.116043 600 raft_consensus.cc:2802] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:50.752060 752 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.159125 467 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.018281 711 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:51.168911 31879 raft_consensus.cc:1238] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
W20250831 07:58:51.200700 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 143532us
I20250831 07:58:51.170387 31881 raft_consensus.cc:2391] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 2: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:51.166594 31773 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.093802 730 consensus_queue.cc:1046] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20250831 07:58:51.203022 730 raft_consensus.cc:3053] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:51.203392 730 raft_consensus.cc:738] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:51.205893 32468 raft_consensus.cc:1238] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.176636 31880 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:51.177006 667 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:51.183118 31878 raft_consensus.cc:1238] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:58:51.179780 31883 raft_consensus.cc:1238] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:51.109550 626 consensus_queue.cc:1046] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20250831 07:58:51.180274 31876 raft_consensus.cc:1238] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.181017 31875 raft_consensus.cc:1238] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.155685 575 raft_consensus.cc:2747] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.186645 31727 raft_consensus.cc:738] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:58:51.189193 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 47698us
I20250831 07:58:51.195556 31877 raft_consensus.cc:1238] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:51.180711 686 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.238198 31730 raft_consensus.cc:1238] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
W20250831 07:58:51.200157 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 72696.6us
I20250831 07:58:51.199990 32434 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.200949 31882 raft_consensus.cc:1238] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.207676 31809 raft_consensus.cc:1238] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.208408 31880 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:51.240716 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:51.241088 31880 raft_consensus.cc:1238] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.241346 31807 raft_consensus.cc:3053] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:51.241701 31807 raft_consensus.cc:738] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:51.242352 31807 consensus_queue.cc:260] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.243137 31807 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.253682 31878 raft_consensus.cc:1238] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:51.254276 31804 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.253886 31879 raft_consensus.cc:1238] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.240648 32434 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
W20250831 07:58:51.257076 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 90563.6us
I20250831 07:58:51.257758 31773 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:51.258056 31807 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:51.180097 656 consensus_queue.cc:260] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.195207 719 consensus_queue.cc:1046] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20250831 07:58:51.253856 31727 consensus_queue.cc:260] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.265619 31727 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.204802 711 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:51.263381 32435 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:51.260905 31849 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.271889 762 consensus_queue.cc:1046] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:51.237701 624 raft_consensus.cc:2747] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:51.257315 683 raft_consensus.cc:3053] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:58:51.281212 31883 raft_consensus.cc:1238] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:51.210079 600 raft_consensus.cc:695] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:51.277817 31876 raft_consensus.cc:1273] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:58:51.279842 31875 raft_consensus.cc:1238] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.253129 714 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.288491 31733 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:51.192137 754 consensus_queue.cc:1046] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20250831 07:58:51.297605 31880 raft_consensus.cc:1273] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250831 07:58:51.285131 743 consensus_meta.cc:287] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.520s user 0.005s sys 0.005s
W20250831 07:58:51.299059 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 58485.4us
I20250831 07:58:51.299787 32434 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.254490 667 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.207178 738 consensus_queue.cc:260] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:51.289362 656 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.316896 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:51.289744 683 raft_consensus.cc:738] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:51.324762 31699 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:51.295282 32465 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:51.290230 730 consensus_queue.cc:260] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.300333 704 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:51.326287 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:58:51.327354 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:51.335505 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:51.336568 31876 raft_consensus.cc:1238] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:51.337159 31881 raft_consensus.cc:1273] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:58:51.307546 736 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.349881 31849 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:51.344008 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 86326.9us
I20250831 07:58:51.352156 31773 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:51.352941 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:51.353467 31875 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.353652 31849 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.309404 743 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.356389 31773 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:51.357244 743 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:51.357910 31805 raft_consensus.cc:1238] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.359174 743 consensus_queue.cc:1046] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:51.355054 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:51.329113 690 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:51.364379 31808 raft_consensus.cc:1238] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:51.326943 767 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:51.329145 759 raft_consensus.cc:1079] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Signalling peer 7ecb257174554a26b2c35ad04b0eabae to start an election
I20250831 07:58:51.355336 598 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.376145 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:51.376369 31729 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:51.378146 667 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:51.380378 771 raft_consensus.cc:2747] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.365921 745 raft_consensus.cc:2747] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.362505 693 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:51.382380 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:51.393090 31879 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:51.344627 772 raft_consensus.cc:2802] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:51.362730 739 raft_consensus.cc:2747] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.328488 763 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:51.374035 737 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250831 07:58:51.403514 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 45858.8us
I20250831 07:58:51.329442 594 consensus_queue.cc:1035] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20250831 07:58:51.364041 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:51.326433 600 consensus_queue.cc:237] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.394467 31698 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:51.397423 770 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:51.370168 690 raft_consensus.cc:695] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:51.370951 642 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:51.368907 731 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:51.396126 31727 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:51.406255 31877 raft_consensus.cc:1273] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:51.402000 742 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:51.408170 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 144860us
I20250831 07:58:51.382284 544 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:51.383723 752 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:51.394181 626 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.419471 32466 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.418603 767 raft_consensus.cc:695] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:51.395006 735 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:51.419986 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:51.388864 31874 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:51.418082 32436 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.430141 32435 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:51.403250 772 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:51.410490 771 raft_consensus.cc:3053] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:51.436476 772 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.436127 31804 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:51.434360 31773 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.442252 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:51.402966 763 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:51.437691 32436 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:51.444629 32434 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.445004 32436 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:51.438810 31875 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
W20250831 07:58:51.446282 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:51.384122 683 consensus_queue.cc:260] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.416661 31731 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:51.410808 686 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.449163 779 raft_consensus.cc:2747] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:58:51.452621 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 97930.7us
I20250831 07:58:51.420682 731 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.437691 771 raft_consensus.cc:738] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:51.447517 763 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.412624 745 consensus_queue.cc:1046] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20250831 07:58:51.460312 31849 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.463886 32436 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:51.465651 31849 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.458400 736 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:51.466562 780 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.459388 683 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:51.473773 31849 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:51.476279 769 raft_consensus.cc:2802] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:51.458148 594 consensus_queue.cc:1035] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20250831 07:58:51.496186 735 consensus_queue.cc:1046] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:51.498258 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:51.499078 613 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:51.500041 31800 raft_consensus.cc:1238] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.501544 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:51.502133 31802 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:51.503057 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:51.504137 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:51.504535 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:58:51.504920 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:51.467324 690 consensus_queue.cc:237] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.476756 642 raft_consensus.cc:2802] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:51.471884 771 consensus_queue.cc:260] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.473412 767 consensus_queue.cc:237] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.516793 738 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:51.490417 575 raft_consensus.cc:2802] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:51.495438 780 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:51.522604 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:51.490630 769 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:51.539096 731 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:51.491762 777 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:51.521281 742 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:51.522449 642 raft_consensus.cc:695] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:51.530030 769 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.531420 575 raft_consensus.cc:695] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:51.536478 780 raft_consensus.cc:695] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:51.553431 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:51.554019 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:51.559197 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:51.541072 556 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:51.554273 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:51.559027 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:51.560405 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:51.550333 31809 raft_consensus.cc:1273] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250831 07:58:51.563791 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:51.574151 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:51.577621 31802 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.573769 32468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 246) took 1044 ms. Trace:
W20250831 07:58:51.585309 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:51.585196 32468 rpcz_store.cc:276] 0831 07:58:50.529373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:50.529773 (+ 400us) service_pool.cc:225] Handling call
0831 07:58:50.530016 (+ 243us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:51.206428 (+676412us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:51.573580 (+367152us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b5400484ed4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:51.573719 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1543700224}
I20250831 07:58:51.535558 754 consensus_queue.cc:1046] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20250831 07:58:51.580363 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:51.592653 31808 raft_consensus.cc:1273] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:58:51.540621 745 consensus_queue.cc:1046] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20250831 07:58:51.580287 766 raft_consensus.cc:3053] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:51.549459 686 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:51.602838 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 139068us
I20250831 07:58:51.559959 32463 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9"
dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:38362
W20250831 07:58:51.610196 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 136525us
I20250831 07:58:51.611995 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:51.612104 31875 raft_consensus.cc:1273] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:58:51.614442 467 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:51.558794 568 consensus_queue.cc:1046] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20250831 07:58:51.615061 467 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:51.561017 736 consensus_queue.cc:1046] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20250831 07:58:51.573123 500 consensus_queue.cc:1035] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20250831 07:58:51.604246 765 raft_consensus.cc:2747] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.613428 734 raft_consensus.cc:3053] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:51.594518 556 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.619359 32463 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:58:51.609498 764 raft_consensus.cc:3053] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:51.546741 703 consensus_queue.cc:1035] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20250831 07:58:51.539620 782 consensus_queue.cc:1046] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.122s
I20250831 07:58:51.614991 766 raft_consensus.cc:738] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:51.591905 775 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.651510 467 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:51.657908 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:51.624006 730 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.668905 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:51.645949 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:51.558622 783 consensus_queue.cc:1046] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.124s
I20250831 07:58:51.674814 31809 raft_consensus.cc:1238] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.668236 544 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:51.675714 544 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.613480 677 consensus_queue.cc:1046] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20250831 07:58:51.649717 734 raft_consensus.cc:738] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:51.678918 771 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.677237 31849 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.659024 32463 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.680799 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:51.660828 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:51.669273 656 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:51.676155 743 raft_consensus.cc:3053] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:51.687662 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:51.617452 770 consensus_queue.cc:1046] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20250831 07:58:51.669176 679 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:51.587311 777 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.664408 778 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:51.694305 31804 raft_consensus.cc:2391] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:51.694430 31882 raft_consensus.cc:2391] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:51.694393 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:51.669816 764 raft_consensus.cc:738] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:51.647259 785 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:51.648921 667 consensus_queue.cc:1035] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20250831 07:58:51.697413 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:51.624940 642 consensus_queue.cc:237] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.646780 575 consensus_queue.cc:237] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.649405 784 raft_consensus.cc:2747] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:51.679334 544 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:51.680413 725 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:51.679476 781 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:51.693334 768 raft_consensus.cc:3053] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:51.693964 743 raft_consensus.cc:738] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:51.710057 556 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:51.702569 568 raft_consensus.cc:2747] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:58:51.683912 774 consensus_queue.cc:1046] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.150s
I20250831 07:58:51.711906 760 raft_consensus.cc:3053] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:51.712373 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:58:51.712919 789 log.cc:927] Time spent T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250831 07:58:51.713359 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:51.730434 31876 raft_consensus.cc:1273] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:51.779140 31878 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:51.730770 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:51.729259 31849 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:51.730454 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:51.732736 31730 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:51.741034 500 consensus_queue.cc:1046] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:58:51.741869 597 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:51.742020 31775 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.745512 31699 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:51.752418 31733 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:51.756789 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:51.760311 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:51.730937 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:51.767901 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:51.698374 656 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.776814 32468 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
W20250831 07:58:51.780565 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 51395.1us
I20250831 07:58:51.705466 679 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:51.786677 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 44737.6us
I20250831 07:58:51.792502 31775 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.793679 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:51.790657 777 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:51.791404 31725 raft_consensus.cc:2391] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:51.795159 31877 raft_consensus.cc:1238] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.802867 31698 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:51.768265 785 raft_consensus.cc:695] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:51.738400 778 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.812659 560 consensus_queue.cc:1035] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:51.773005 768 raft_consensus.cc:738] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:51.810181 31775 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.779665 751 raft_consensus.cc:3053] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Stepping down as leader of term 2
W20250831 07:58:51.812768 791 log.cc:927] Time spent T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.131s user 0.001s sys 0.000s
I20250831 07:58:51.812988 747 raft_consensus.cc:3053] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
W20250831 07:58:51.809619 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 64216.4us
I20250831 07:58:51.836907 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:51.837491 31733 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:51.782629 568 raft_consensus.cc:2747] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.793493 781 raft_consensus.cc:695] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:51.817770 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:51.818627 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:51.821484 31728 raft_consensus.cc:1238] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.822283 31734 raft_consensus.cc:1238] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:51.791720 782 raft_consensus.cc:2747] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.813580 736 raft_consensus.cc:2747] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.785063 760 raft_consensus.cc:738] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:51.807477 31729 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:51.817103 31805 raft_consensus.cc:1273] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:51.771857 725 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.668776 780 consensus_queue.cc:237] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.808357 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:51.859795 31882 raft_consensus.cc:1238] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:51.855216 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:51.842048 747 raft_consensus.cc:738] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:51.831419 31775 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.829654 672 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:51.815810 544 raft_consensus.cc:2747] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.837349 774 raft_consensus.cc:2747] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.849964 561 consensus_queue.cc:1035] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20250831 07:58:51.871891 656 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:51.872123 731 raft_consensus.cc:3053] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:51.849468 505 consensus_queue.cc:1035] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20250831 07:58:51.792186 597 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.832823 751 raft_consensus.cc:738] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:51.841828 752 consensus_queue.cc:1046] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 2, Time since last communication: 0.036s
I20250831 07:58:51.829988 743 consensus_queue.cc:260] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.890906 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:51.892096 31732 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:51.897662 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 58491.1us
I20250831 07:58:51.851181 32463 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.882778 672 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:58:51.881275 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 49980.4us
I20250831 07:58:51.919361 31732 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:51.768448 764 consensus_queue.cc:260] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.884410 736 raft_consensus.cc:2747] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:58:51.915004 796 log.cc:927] Time spent T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.154s user 0.000s sys 0.002s
I20250831 07:58:51.915879 597 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:51.916090 32463 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:51.909441 31731 raft_consensus.cc:2391] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 2: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
W20250831 07:58:51.906256 711 consensus_meta.cc:287] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.614s user 0.005s sys 0.006s
I20250831 07:58:51.920665 31775 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:51.908728 577 consensus_meta.cc:287] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.783s user 0.007s sys 0.004s
I20250831 07:58:51.898802 731 raft_consensus.cc:738] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:51.882314 31733 raft_consensus.cc:2466] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:51.922274 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:51.922282 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:51.887058 719 consensus_queue.cc:1046] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20250831 07:58:51.924517 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:51.900895 760 consensus_queue.cc:260] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.934262 31725 raft_consensus.cc:1238] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:51.936205 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:51.921335 672 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:51.905041 544 consensus_queue.cc:1046] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20250831 07:58:51.941900 31882 raft_consensus.cc:2391] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:51.943217 31775 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.943369 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:51.951661 32468 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
W20250831 07:58:51.952603 793 log.cc:927] Time spent T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.214s user 0.001s sys 0.000s
I20250831 07:58:51.951680 31775 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.958642 31775 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:51.959604 656 raft_consensus.cc:2747] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.916579 781 consensus_queue.cc:237] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:51.959942 731 consensus_queue.cc:260] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.961326 731 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:51.937837 679 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:51.963171 760 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:51.964670 31698 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:51.965728 31775 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:51.975020 31775 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:51.995162 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 108893us
I20250831 07:58:51.945679 736 raft_consensus.cc:2747] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.996074 31699 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:51.995393 704 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:51.964540 688 consensus_meta.cc:287] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.869s user 0.012s sys 0.000s
I20250831 07:58:51.954840 752 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:51.937768 711 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.954254 677 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:51.906769 768 consensus_queue.cc:260] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.945163 747 consensus_queue.cc:260] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.967590 782 raft_consensus.cc:2747] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:51.998409 777 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:51.939599 577 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:51.968896 656 raft_consensus.cc:2802] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:51.947687 783 consensus_queue.cc:1046] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20250831 07:58:52.024698 783 raft_consensus.cc:2747] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:52.025004 774 consensus_queue.cc:1046] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20250831 07:58:52.025720 774 consensus_queue.cc:1230] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Peer a0d845d01c8c4894895f66e3beae298c log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250831 07:58:52.026700 577 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:52.027513 31804 raft_consensus.cc:2391] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:52.027712 744 raft_consensus.cc:3053] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:52.028240 744 raft_consensus.cc:738] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:52.029049 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:52.029666 31804 raft_consensus.cc:3053] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:52.029959 31804 raft_consensus.cc:738] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:52.035266 753 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:51.877110 785 consensus_queue.cc:237] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.044142 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:51.921150 751 consensus_queue.cc:260] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.007205 688 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.012105 752 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.036962 711 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:52.036792 656 raft_consensus.cc:695] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:52.047505 31804 consensus_queue.cc:260] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.019304 677 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.066256 31729 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:52.068353 31877 raft_consensus.cc:1273] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:58:52.068768 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:52.046303 778 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:52.042021 753 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:52.061853 763 consensus_meta.cc:287] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.588s user 0.012s sys 0.001s
I20250831 07:58:52.051514 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:52.075500 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:58:52.090384 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 94394.9us
I20250831 07:58:52.053640 803 raft_consensus.cc:2747] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:52.052204 725 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:52.092712 725 consensus_queue.cc:1035] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:58:52.029229 744 consensus_queue.cc:260] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.071879 751 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:52.121366 656 consensus_queue.cc:237] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.122896 753 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:52.125079 31876 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:52.180577 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:52.089951 31803 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
W20250831 07:58:52.180771 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:52.185931 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 86490.3us
W20250831 07:58:52.188985 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 84830.4us
I20250831 07:58:52.180588 561 raft_consensus.cc:3053] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:52.224082 561 raft_consensus.cc:738] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:52.224989 561 consensus_queue.cc:260] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.093591 677 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:52.226799 561 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:52.227882 790 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:52.228266 624 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:52.228945 624 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:52.230927 806 log.cc:927] Time spent T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20250831 07:58:52.231132 624 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:52.228209 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:52.241956 31875 raft_consensus.cc:1273] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250831 07:58:52.221915 775 consensus_meta.cc:287] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.501s user 0.003s sys 0.007s
I20250831 07:58:52.249279 775 raft_consensus.cc:2747] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:52.251549 792 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:58:52.235019 744 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:52.272426 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:52.091480 763 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.091444 31699 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:58:52.304980 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 213638us
I20250831 07:58:52.305601 32465 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:52.306478 763 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:52.307250 31849 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
W20250831 07:58:52.245383 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 276810us
I20250831 07:58:52.228477 790 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.310366 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:52.280144 688 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:52.331434 683 consensus_meta.cc:287] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.855s user 0.012s sys 0.000s
I20250831 07:58:52.324189 31775 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:58:52.308436 772 consensus_meta.cc:287] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.858s user 0.011s sys 0.001s
I20250831 07:58:52.348284 772 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.350529 31800 raft_consensus.cc:1238] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:52.353410 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:52.355719 772 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:52.359393 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:52.379328 31807 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
W20250831 07:58:52.397337 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 73279.8us
I20250831 07:58:52.398481 31775 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:58:52.312913 769 consensus_meta.cc:287] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.739s user 0.005s sys 0.007s
I20250831 07:58:52.400578 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:52.316655 629 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:52.321125 803 raft_consensus.cc:2747] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:58:52.320364 782 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:52.417691 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:52.407855 688 raft_consensus.cc:2802] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:52.412871 683 raft_consensus.cc:2802] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:52.486131 31878 raft_consensus.cc:1238] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:52.418347 752 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:52.417443 629 raft_consensus.cc:695] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:52.475742 690 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:52.489491 672 consensus_meta.cc:287] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.525s user 0.007s sys 0.003s
W20250831 07:58:52.511030 704 consensus_meta.cc:287] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.511s user 0.011s sys 0.000s
I20250831 07:58:52.488993 688 raft_consensus.cc:695] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:52.544363 32469 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:52.544732 779 raft_consensus.cc:3053] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:52.408272 769 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.549183 629 consensus_queue.cc:237] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.554343 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1415) took 1595 ms. Trace:
I20250831 07:58:52.510097 683 raft_consensus.cc:695] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:52.475839 782 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.525382 677 consensus_queue.cc:1046] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20250831 07:58:52.571789 753 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:52.563117 31803 rpcz_store.cc:276] 0831 07:58:50.958554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:50.975701 (+ 17147us) service_pool.cc:225] Handling call
0831 07:58:51.149685 (+173984us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:52.227347 (+1077662us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:52.554057 (+326710us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x7b540009e554. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:52.554272 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2105331456}
I20250831 07:58:52.540339 704 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:52.594250 753 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.595016 600 raft_consensus.cc:3053] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:52.595429 600 raft_consensus.cc:738] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:52.596199 600 consensus_queue.cc:260] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.596429 753 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:52.597657 735 raft_consensus.cc:3053] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:52.597997 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:52.598114 735 raft_consensus.cc:738] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:52.604504 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1314) took 1105 ms. Trace:
I20250831 07:58:52.560583 779 raft_consensus.cc:738] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:52.604884 31800 rpcz_store.cc:276] 0831 07:58:51.498626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:51.502047 (+ 3421us) service_pool.cc:225] Handling call
0831 07:58:51.502420 (+ 373us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:52.351082 (+848662us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:52.604270 (+253188us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b5400028d54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:52.604451 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1949890816}
I20250831 07:58:52.529462 690 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.536206 672 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:52.600756 738 consensus_meta.cc:287] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 1.084s user 0.013s sys 0.002s
I20250831 07:58:52.683677 778 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:52.688916 31808 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
W20250831 07:58:52.722486 556 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.008s sys 0.081s
I20250831 07:58:52.722721 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1307) took 1191 ms. Trace:
I20250831 07:58:52.722807 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:52.723292 31807 rpcz_store.cc:276] 0831 07:58:51.496675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:51.503904 (+ 7229us) service_pool.cc:225] Handling call
0831 07:58:52.687867 (+1183963us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b54000ce4d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:52.687988 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1903040256}
W20250831 07:58:52.723088 556 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s user 0.008s sys 0.082s
I20250831 07:58:52.684793 753 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:52.694806 772 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:52.754084 688 consensus_queue.cc:237] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.769212 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1426) took 1043 ms. Trace:
I20250831 07:58:52.745095 790 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:52.684475 778 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.783942 31878 raft_consensus.cc:1273] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:52.769762 31803 rpcz_store.cc:276] 0831 07:58:51.725792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.724207 (+998415us) service_pool.cc:225] Handling call
0831 07:58:52.744310 (+ 20103us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:52.758524 (+ 14214us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:52.760754 (+ 2230us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:52.768783 (+ 8029us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:58:52.768855 (+ 72us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:52.768956 (+ 101us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:52.769126 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39296}
I20250831 07:58:52.790777 766 consensus_queue.cc:260] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.797662 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 70) took 1046 ms. Trace:
I20250831 07:58:52.752969 824 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:52.755899 735 consensus_queue.cc:260] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.767315 772 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:52.771014 753 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.798224 31803 rpcz_store.cc:276] 0831 07:58:51.751763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.790688 (+1038925us) service_pool.cc:225] Handling call
0831 07:58:52.791113 (+ 425us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:52.794725 (+ 3612us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:52.794782 (+ 57us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:52.794858 (+ 76us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:58:52.794937 (+ 79us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:52.795028 (+ 91us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:52.797586 (+ 2558us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:52.802479 760 consensus_meta.cc:287] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.839s user 0.000s sys 0.012s
I20250831 07:58:52.814615 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:52.806738 771 consensus_meta.cc:287] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 1.112s user 0.013s sys 0.000s
I20250831 07:58:52.819482 31809 raft_consensus.cc:1273] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:58:52.803930 735 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:52.798964 824 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.795732 719 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20250831 07:58:52.767519 683 consensus_queue.cc:237] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.827255 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1326) took 1159 ms. Trace:
I20250831 07:58:52.792416 779 consensus_queue.cc:260] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.827659 31805 rpcz_store.cc:276] 0831 07:58:51.668085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:51.671770 (+ 3685us) service_pool.cc:225] Handling call
0831 07:58:51.672051 (+ 281us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:51.870355 (+198304us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:52.825111 (+954756us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b540015ac54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:52.827196 (+ 2085us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":332828416}
I20250831 07:58:52.833906 31800 raft_consensus.cc:1238] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
W20250831 07:58:52.834015 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:52.834575 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1425) took 1126 ms. Trace:
I20250831 07:58:52.834991 31800 rpcz_store.cc:276] 0831 07:58:51.708529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.833323 (+1124794us) service_pool.cc:225] Handling call
0831 07:58:52.833725 (+ 402us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:52.834333 (+ 608us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:52.834512 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:52.836447 32471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38366 (request call id 344) took 1155 ms. Trace:
I20250831 07:58:52.836910 32471 rpcz_store.cc:276] 0831 07:58:51.680813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:51.680934 (+ 121us) service_pool.cc:225] Handling call
0831 07:58:52.836169 (+1155235us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b54004c13d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:52.836329 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":872809728}
I20250831 07:58:52.839418 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1429) took 1069 ms. Trace:
I20250831 07:58:52.840003 31800 rpcz_store.cc:276] 0831 07:58:51.770454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.838595 (+1068141us) service_pool.cc:225] Handling call
0831 07:58:52.838947 (+ 352us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:52.839113 (+ 166us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:52.839137 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:52.839178 (+ 41us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:58:52.839207 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:52.839267 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:52.839367 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:52.841550 31800 raft_consensus.cc:1238] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:52.842736 31800 raft_consensus.cc:1238] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:52.849151 747 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:52.853119 733 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:52.853672 733 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:52.859076 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:52.860255 568 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:52.866199 568 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.866874 778 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:52.870026 31808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1424) took 1217 ms. Trace:
I20250831 07:58:52.870491 31808 rpcz_store.cc:276] 0831 07:58:51.652403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:51.652513 (+ 110us) service_pool.cc:225] Handling call
0831 07:58:52.869848 (+1217335us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54001d8b54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:52.869968 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2336138112}
I20250831 07:58:52.870993 760 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:52.871230 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:52.872745 31807 raft_consensus.cc:1273] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250831 07:58:52.872853 31849 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.020s sys 0.094s
W20250831 07:58:52.880155 31849 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.564s user 0.020s sys 0.095s
I20250831 07:58:52.871765 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:58:52.881242 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 565437us
W20250831 07:58:52.886847 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:52.851969 769 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:52.870180 568 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:52.816285 782 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:52.900590 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:52.901571 824 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:52.903292 600 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:52.871222 32469 raft_consensus.cc:1273] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:52.894459 690 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:52.904745 31806 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:52.873113 672 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:52.905966 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1427) took 1147 ms. Trace:
W20250831 07:58:52.910073 821 log.cc:927] Time spent T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20250831 07:58:52.908444 790 consensus_queue.cc:1046] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20250831 07:58:52.904111 568 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:52.910102 31803 rpcz_store.cc:276] 0831 07:58:51.758226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.824054 (+1065828us) service_pool.cc:225] Handling call
0831 07:58:52.899097 (+ 75043us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:52.904070 (+ 4973us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:52.904102 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:52.904171 (+ 69us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:52.904212 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:52.905767 (+ 1555us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:52.905906 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:52.912688 568 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.913777 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:52.914942 568 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:52.915133 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:52.915374 32435 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:52.917449 31848 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:52.917515 467 consensus_queue.cc:1046] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:52.916133 824 consensus_queue.cc:1046] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:52.921386 677 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:52.921689 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:52.922325 31876 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:58:52.922611 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1329) took 1244 ms. Trace:
I20250831 07:58:52.923004 31809 rpcz_store.cc:276] 0831 07:58:51.678094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:51.680073 (+ 1979us) service_pool.cc:225] Handling call
0831 07:58:51.681798 (+ 1725us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:52.820020 (+1138222us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:52.922376 (+102356us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540009e054. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:52.922552 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2615550720}
I20250831 07:58:52.924155 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:52.924949 31809 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
W20250831 07:58:52.926599 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:52.928181 32434 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:52.927304 31848 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:58:52.932330 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:52.932714 788 consensus_queue.cc:1046] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20250831 07:58:52.933425 790 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:58:52.933815 790 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:52.934296 788 raft_consensus.cc:3053] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:52.934242 467 raft_consensus.cc:3053] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:52.938871 467 raft_consensus.cc:738] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:52.934952 31882 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:52.940398 31878 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:52.872143 760 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:52.937561 824 raft_consensus.cc:3053] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:52.934201 790 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:52.934603 571 raft_consensus.cc:3053] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:52.939669 467 consensus_queue.cc:260] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:52.945482 730 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:58:52.907459 738 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:52.871572 742 consensus_queue.cc:752] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 2 is ahead of the local log (next sequential op: 2). Destination peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 1.095s
I20250831 07:58:52.923767 779 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:52.956758 467 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:52.946672 677 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:58:52.990463 32465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 249) took 1405 ms. Trace:
I20250831 07:58:52.990854 677 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:52.991407 677 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:52.991991 781 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:52.924305 31803 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:52.938696 788 raft_consensus.cc:738] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:52.946301 824 raft_consensus.cc:738] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:52.952623 730 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:58:52.992731 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:52.953737 571 raft_consensus.cc:738] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:53.013801 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:53.016508 31848 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:53.011945 739 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.029131 31878 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:53.036911 32434 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:58:53.030491 597 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.011s sys 0.032s
I20250831 07:58:52.957597 763 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.075s
I20250831 07:58:53.029237 730 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:58:52.957386 738 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.014580 733 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:53.014299 626 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.032249 32469 raft_consensus.cc:1238] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:53.052155 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:53.053332 773 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:53.053186 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:53.039139 753 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:53.035333 782 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:58:53.085103 31698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.843s user 0.080s sys 0.119s
W20250831 07:58:53.086619 31698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.845s user 0.081s sys 0.119s
W20250831 07:58:53.087752 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 845782us
I20250831 07:58:53.005218 781 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:53.063980 561 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.013s sys 0.021s
W20250831 07:58:53.024835 31846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.002s sys 0.024s
W20250831 07:58:53.101460 31846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.644s user 0.003s sys 0.024s
W20250831 07:58:53.102226 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:53.046979 597 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s user 0.011s sys 0.032s
I20250831 07:58:53.044831 767 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:53.038642 747 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:52.991134 32465 rpcz_store.cc:276] 0831 07:58:51.549403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:51.580747 (+ 31344us) service_pool.cc:225] Handling call
0831 07:58:51.644719 (+ 63972us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:52.306118 (+661399us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:52.944160 (+638042us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b540074e354. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:52.954294 (+ 10134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1518998912}
W20250831 07:58:53.069453 31699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.750s user 0.005s sys 0.014s
I20250831 07:58:52.957808 734 consensus_queue.cc:260] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:53.047585 731 consensus_meta.cc:287] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 1.086s user 0.014s sys 0.000s
W20250831 07:58:53.097330 31700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.843s user 0.006s sys 0.020s
W20250831 07:58:53.103200 561 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.795s user 0.013s sys 0.022s
W20250831 07:58:53.125378 31700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.871s user 0.006s sys 0.020s
W20250831 07:58:53.063572 31875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.009s sys 0.011s
I20250831 07:58:53.049278 774 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.038565 739 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.107533 767 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:53.073316 626 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.114073 745 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:53.116024 31699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.797s user 0.005s sys 0.014s
I20250831 07:58:53.114526 467 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.144658 467 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.123445 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1311) took 1625 ms. Trace:
I20250831 07:58:53.145792 31806 rpcz_store.cc:276] 0831 07:58:51.497819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:51.504307 (+ 6488us) service_pool.cc:225] Handling call
0831 07:58:53.123221 (+1618914us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54001d9a54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:53.123372 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3037260032}
I20250831 07:58:53.100433 782 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:53.133874 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:53.131516 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:53.053218 819 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:53.121920 751 consensus_meta.cc:287] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.939s user 0.008s sys 0.006s
I20250831 07:58:53.111783 760 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:53.113507 747 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:53.134583 830 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.137250 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:53.099812 788 consensus_queue.cc:260] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.113734 826 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.090629 824 consensus_queue.cc:260] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.140677 767 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:53.117360 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:53.137833 731 consensus_queue.cc:1046] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20250831 07:58:53.115676 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:53.136989 31877 raft_consensus.cc:1273] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:53.100945 753 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:53.129695 805 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.150171 32470 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 3
I20250831 07:58:53.151286 31805 raft_consensus.cc:1238] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:53.130661 736 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.152243 31806 raft_consensus.cc:1273] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:53.152803 467 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:53.141048 772 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
W20250831 07:58:53.148236 31697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.035s sys 0.049s
W20250831 07:58:53.067603 656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.056s sys 0.060s
I20250831 07:58:53.106489 571 consensus_queue.cc:260] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.167212 830 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.169044 31879 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:53.106779 773 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:53.129894 31875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s user 0.009s sys 0.011s
I20250831 07:58:53.159960 747 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:58:53.181294 32436 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.015s sys 0.031s
I20250831 07:58:53.137682 774 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:53.194309 31698 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:58:53.165656 803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.014s sys 0.014s
I20250831 07:58:53.173416 823 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:58:53.210053 32436 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s user 0.015s sys 0.035s
I20250831 07:58:53.192374 788 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:53.214536 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1335) took 1486 ms. Trace:
I20250831 07:58:53.162933 819 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.198549 672 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:58:53.205519 31697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.810s user 0.036s sys 0.049s
W20250831 07:58:53.202646 656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s user 0.056s sys 0.061s
I20250831 07:58:53.220186 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:53.220999 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:58:53.219290 803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s user 0.014s sys 0.014s
I20250831 07:58:53.221727 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:53.223553 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:53.198812 686 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:53.224419 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:53.148746 745 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.226792 31807 rpcz_store.cc:276] 0831 07:58:51.728395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.724207 (+995812us) service_pool.cc:225] Handling call
0831 07:58:52.872342 (+148135us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:52.879094 (+ 6752us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:53.214272 (+335178us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54001d8b50. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:53.214460 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":339812224}
I20250831 07:58:53.227666 31879 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:58:53.190174 568 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.228870 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:53.451473 31725 raft_consensus.cc:2391] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:53.221052 467 consensus_queue.cc:1035] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250831 07:58:53.260341 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:53.267017 31732 raft_consensus.cc:2391] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 2: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
W20250831 07:58:53.237228 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 119150us
W20250831 07:58:53.272572 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 78367.1us
I20250831 07:58:53.229069 824 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:53.232349 810 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.196689 753 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:53.201365 736 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.225215 505 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:58:53.288889 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:53.207468 772 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.232534 825 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.284242 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:53.291152 781 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:53.235356 571 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:53.460942 768 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:53.317404 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:53.275935 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:53.167941 826 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.242552 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:53.324292 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1378) took 1220 ms. Trace:
I20250831 07:58:53.235996 703 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:53.266641 751 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.266685 629 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:53.289633 822 log.cc:927] Time spent T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.182s user 0.000s sys 0.001s
I20250831 07:58:53.199093 805 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:53.337669 774 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:53.280462 597 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:53.288148 804 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:53.283682 624 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:53.344072 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:53.351233 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:53.267643 672 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:53.322664 769 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.431224 31699 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:58:53.438498 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:53.571902 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:53.572223 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 141248us
I20250831 07:58:53.440646 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:53.445806 31775 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:53.574077 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 128361us
I20250831 07:58:53.448559 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:53.576184 31733 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:53.577306 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:53.577986 31733 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:53.456710 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:53.429956 31729 raft_consensus.cc:1238] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:53.459693 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:53.346230 626 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:53.233067 823 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:53.461596 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:53.470288 31698 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:58:53.472656 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:53.445457 782 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:53.461553 845 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.449224 739 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:53.343073 836 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:53.478109 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:53.235349 828 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.476810 594 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.489810 31734 raft_consensus.cc:1238] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:53.491956 31882 rpcz_store.cc:276] 0831 07:58:52.068336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.068542 (+ 206us) service_pool.cc:225] Handling call
0831 07:58:53.288802 (+1220260us) spinlock_profiling.cc:243] Waited 865 ms on lock 0x7b540020cbd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:53.288909 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1990596864}
W20250831 07:58:53.493175 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:53.580930 803 raft_consensus.cc:2747] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:53.451627 686 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:53.480000 556 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.479312 829 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:53.459353 568 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.481695 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:53.563529 745 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:53.566551 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:53.653265 31733 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
W20250831 07:58:53.623644 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:53.654034 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:53.667652 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 47869.2us
I20250831 07:58:53.466524 810 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:53.550334 841 log.cc:927] Time spent T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.208s user 0.001s sys 0.000s
I20250831 07:58:53.471302 505 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:53.552311 600 consensus_meta.cc:287] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.649s user 0.010s sys 0.001s
W20250831 07:58:53.570667 832 log.cc:927] Time spent T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.393s user 0.000s sys 0.001s
I20250831 07:58:53.334892 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:53.580127 31775 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:53.581363 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1428) took 1816 ms. Trace:
I20250831 07:58:53.725375 31805 rpcz_store.cc:276] 0831 07:58:51.764805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.828608 (+1063803us) service_pool.cc:225] Handling call
0831 07:58:52.828905 (+ 297us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:53.190671 (+361766us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:53.581123 (+390452us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b5400038ed4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:53.581305 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":740938496}
I20250831 07:58:53.670508 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:53.553730 31726 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:53.474478 772 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.592190 32470 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:58:53.732509 31730 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:58:53.480831 825 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:53.602150 31732 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:53.568475 774 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.613898 31773 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:53.510821 467 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.554618 751 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.556310 629 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.740599 467 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.590086 575 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.553580 703 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.569563 819 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:53.581003 773 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:53.603065 667 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250831 07:58:53.666761 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:53.683549 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:53.751515 667 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:53.754308 853 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:53.620559 561 consensus_queue.cc:1035] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.122s
I20250831 07:58:53.579613 624 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.572973 597 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.669634 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:53.666356 556 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:53.701067 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:53.764029 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:53.712291 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:53.589465 769 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:53.723420 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 150944us
I20250831 07:58:53.670076 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:53.768795 31879 raft_consensus.cc:2391] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:53.597023 544 consensus_queue.cc:1046] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.090s
I20250831 07:58:53.597957 626 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:58:53.760361 835 log.cc:927] Time spent T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.146s user 0.000s sys 0.000s
W20250831 07:58:53.827036 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 58426.3us
W20250831 07:58:53.729744 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 153914us
I20250831 07:58:53.828594 31775 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:53.830047 31775 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:53.594164 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 124003us
I20250831 07:58:53.737954 860 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.738411 736 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:53.738664 686 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:53.739236 505 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:53.683586 829 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:53.740831 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 127273us
I20250831 07:58:53.840129 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:53.840219 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:53.841205 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:53.618822 814 consensus_queue.cc:1035] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:53.713454 766 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:53.654430 858 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.734680 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:53.855911 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:53.856377 31805 raft_consensus.cc:2391] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:53.711652 745 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.750941 859 raft_consensus.cc:2747] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:53.751490 467 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:53.736624 735 consensus_meta.cc:287] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.884s user 0.009s sys 0.002s
I20250831 07:58:53.739449 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:53.687618 803 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:53.728998 827 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.728955 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:53.619273 833 consensus_queue.cc:1035] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.080s
I20250831 07:58:53.754603 667 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:53.732586 782 raft_consensus.cc:2747] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:58:53.617106 845 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.734138 784 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.620715 627 consensus_queue.cc:1035] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.126s
I20250831 07:58:53.759373 805 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:53.760787 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1337) took 1438 ms. Trace:
W20250831 07:58:53.893716 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:53.733213 851 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:53.620126 594 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.761606 31877 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:53.616712 828 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.765352 31727 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:53.615075 790 consensus_meta.cc:287] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.622s user 0.010s sys 0.001s
I20250831 07:58:53.770968 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 74) took 1516 ms. Trace:
I20250831 07:58:53.753396 753 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:53.836485 853 consensus_queue.cc:1046] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 2, Time since last communication: 0.097s
I20250831 07:58:53.842995 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:53.771241 839 log.cc:927] Time spent T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20250831 07:58:53.749373 774 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.858840 823 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:53.752468 575 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:53.865021 844 log.cc:927] Time spent T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.369s user 0.001s sys 0.000s
I20250831 07:58:53.836108 773 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:53.616901 836 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:53.870857 467 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.868448 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:53.872854 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:53.840464 736 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.876456 31807 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:53.878404 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:53.879501 850 log.cc:927] Time spent T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.301s user 0.000s sys 0.002s
I20250831 07:58:53.880064 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:54.132002 736 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:53.880322 842 log.cc:927] Time spent T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.383s user 0.001s sys 0.000s
I20250831 07:58:53.880803 857 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:53.881383 840 log.cc:927] Time spent T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.417s user 0.000s sys 0.001s
I20250831 07:58:53.882848 743 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:53.886493 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:54.081729 866 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:53.865962 837 log.cc:927] Time spent T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.265s user 0.001s sys 0.000s
W20250831 07:58:53.869310 838 log.cc:927] Time spent T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.334s user 0.000s sys 0.001s
I20250831 07:58:53.895777 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:53.856555 600 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:53.836931 626 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:53.899242 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1433) took 1040 ms. Trace:
I20250831 07:58:53.927589 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:53.901834 31849 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:54.169267 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 267493us
W20250831 07:58:54.026433 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:54.169462 31806 rpcz_store.cc:276] 0831 07:58:52.859105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.195476 (+336371us) service_pool.cc:225] Handling call
0831 07:58:53.209322 (+ 13846us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:53.232643 (+ 23321us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:53.232688 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:53.232754 (+ 66us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:53.232799 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:53.899049 (+666250us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b54000cf3d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:53.899091 (+ 42us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:53.899192 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":53033728}
W20250831 07:58:54.028460 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:54.031334 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 178165us
I20250831 07:58:54.024888 31809 rpcz_store.cc:276] 0831 07:58:52.322875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.934558 (+611683us) service_pool.cc:225] Handling call
0831 07:58:52.937322 (+ 2764us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:52.937632 (+ 310us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:52.937662 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:58:52.938111 (+ 449us) log.cc:844] Serialized 26 byte log entry
0831 07:58:53.143937 (+205826us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:58:53.144181 (+ 244us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:53.144332 (+ 151us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:58:53.760581 (+616249us) raft_consensus.cc:1683] finished
0831 07:58:53.760603 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:53.760729 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":443,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"thread_start_us":203715,"threads_started":1,"wal-append.queue_time_us":253049}
I20250831 07:58:53.894202 854 raft_consensus.cc:2747] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:53.853631 819 consensus_queue.cc:1046] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20250831 07:58:53.893200 771 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:53.897433 767 consensus_meta.cc:287] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.717s user 0.010s sys 0.000s
I20250831 07:58:54.174228 31698 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:53.877681 672 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:53.886546 735 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.886755 826 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:53.900034 781 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.857003 860 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.928051 833 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:53.871248 745 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:53.841739 31775 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:54.212543 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 370858us
I20250831 07:58:54.213784 31775 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:54.214768 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:54.215265 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:54.050854 31875 rpcz_store.cc:276] 0831 07:58:52.254264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.254405 (+ 141us) service_pool.cc:225] Handling call
0831 07:58:52.302271 (+ 47866us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:52.406599 (+104328us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:52.406651 (+ 52us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:58:52.413438 (+ 6787us) log.cc:844] Serialized 26 byte log entry
0831 07:58:53.206535 (+793097us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:58:53.206590 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:53.206680 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:58:53.770804 (+564124us) raft_consensus.cc:1683] finished
0831 07:58:53.770825 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:53.770924 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1141,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"thread_start_us":791167,"threads_started":2,"wal-append.queue_time_us":650981}
W20250831 07:58:54.063512 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:54.051374 683 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:54.052672 568 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:54.106387 825 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:54.067198 739 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:54.114490 31725 raft_consensus.cc:2391] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:54.108315 467 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.191004 805 raft_consensus.cc:2747] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:58:54.129820 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 48375.1us
I20250831 07:58:54.133219 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 81) took 1194 ms. Trace:
I20250831 07:58:54.134143 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:54.138154 751 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:54.110929 853 consensus_queue.cc:1230] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Peer a0d845d01c8c4894895f66e3beae298c log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 07:58:54.093925 764 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:54.064388 790 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:54.207644 672 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:54.351790 31875 raft_consensus.cc:1273] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:54.057279 784 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.116726 629 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:54.160533 862 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:54.126434 772 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.188426 823 consensus_queue.cc:1035] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20250831 07:58:54.136752 703 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.149644 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:58:54.177544 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:54.371984 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:54.198544 767 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:54.083773 851 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.154536 753 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:54.196166 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:54.053964 827 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.187544 836 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.201236 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:54.201396 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:53.878849 858 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.202603 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:54.362596 672 raft_consensus.cc:695] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:54.166450 769 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.153642 624 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:54.188133 849 log.cc:927] Time spent T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.364s user 0.000s sys 0.001s
I20250831 07:58:54.169323 597 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.218351 845 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:53.886185 803 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.178035 859 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:54.221980 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:54.178156 782 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:54.237601 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 990) took 1014 ms. Trace:
I20250831 07:58:54.403546 31725 rpcz_store.cc:276] 0831 07:58:53.222741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.854845 (+632104us) service_pool.cc:225] Handling call
0831 07:58:54.237530 (+382685us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:54.240418 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1336) took 1918 ms. Trace:
I20250831 07:58:54.155938 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:54.405249 31734 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:54.230937 31876 rpcz_store.cc:276] 0831 07:58:52.939302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.940888 (+ 1586us) service_pool.cc:225] Handling call
0831 07:58:52.954718 (+ 13830us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:53.004377 (+ 49659us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:53.004425 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:58:53.028397 (+ 23972us) log.cc:844] Serialized 26 byte log entry
0831 07:58:53.189492 (+161095us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:58:53.189547 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:53.189638 (+ 91us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:58:54.132995 (+943357us) raft_consensus.cc:1683] finished
0831 07:58:54.133017 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:54.133151 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10908,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":19284,"thread_start_us":134825,"threads_started":1,"wal-append.queue_time_us":403496}
I20250831 07:58:54.247805 467 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:54.267294 31727 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:54.354115 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:54.221776 854 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:54.189556 600 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.126199 556 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:54.366451 31883 raft_consensus.cc:2391] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:54.198697 771 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.204829 735 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.173655 810 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:54.363922 860 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:54.350555 853 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:54.177058 31848 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:54.352912 760 raft_consensus.cc:3053] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:54.351541 736 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.374401 594 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.424017 782 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.443460 858 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:54.401302 774 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.446712 672 consensus_queue.cc:237] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:54.439484 760 raft_consensus.cc:738] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:54.444394 860 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:54.377625 31881 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:54.357676 627 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:54.370074 677 consensus_meta.cc:287] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 1.378s user 0.012s sys 0.001s
I20250831 07:58:54.382299 865 raft_consensus.cc:2747] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 07:58:54.456039 779 consensus_meta.cc:287] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 1.453s user 0.008s sys 0.006s
I20250831 07:58:54.229171 683 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:54.367312 31874 raft_consensus.cc:1273] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:54.389633 31809 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
W20250831 07:58:54.392606 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:54.240453 739 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:54.370766 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:54.178382 857 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.377796 703 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:54.440141 859 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.377589 31731 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:54.365057 867 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:54.380699 575 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:54.411900 31879 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:54.370774 862 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:54.364634 829 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:54.430056 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:54.451947 853 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.441951 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:54.404247 31800 rpcz_store.cc:276] 0831 07:58:52.322521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.872769 (+550248us) service_pool.cc:225] Handling call
0831 07:58:52.896120 (+ 23351us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:52.896888 (+ 768us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:52.896915 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:58:52.905591 (+ 8676us) log.cc:844] Serialized 26 byte log entry
0831 07:58:53.202575 (+296984us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:58:53.202617 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:53.202691 (+ 74us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:58:54.240153 (+1037462us) raft_consensus.cc:1683] finished
0831 07:58:54.240191 (+ 38us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:54.240350 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3163,"raft.queue_time_us":144,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":4438,"thread_start_us":254427,"threads_started":1,"wal-append.queue_time_us":352208}
I20250831 07:58:54.571091 31809 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
mode: GRACEFUL
new_leader_uuid: "8247a38b550649aa9cd6e7668dcaec10"
from {username='slave'} at 127.0.0.1:37044
I20250831 07:58:54.206684 781 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:54.386646 753 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:54.378582 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 186143us
I20250831 07:58:54.583069 31698 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:54.168773 866 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.461125 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:54.384709 872 raft_consensus.cc:2747] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:54.240965 833 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:54.517477 32471 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:54.471652 627 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:54.471957 31729 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
W20250831 07:58:54.522691 31726 consensus_meta.cc:287] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.771s user 0.002s sys 0.007s
W20250831 07:58:54.523622 788 consensus_meta.cc:287] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.256s user 0.006s sys 0.006s
W20250831 07:58:54.469937 730 consensus_meta.cc:287] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.405s user 0.009s sys 0.003s
W20250831 07:58:54.469117 667 consensus_meta.cc:287] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.577s user 0.009s sys 0.000s
W20250831 07:58:54.541453 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:54.546097 31880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 84) took 1415 ms. Trace:
I20250831 07:58:54.623062 739 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:54.514382 626 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:54.548723 703 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.563298 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:54.623585 739 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:54.621541 767 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.624246 739 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.624894 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:54.626127 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:54.627226 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:54.572628 828 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:54.628057 824 consensus_meta.cc:287] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.159s user 0.011s sys 0.000s
I20250831 07:58:54.549584 31882 raft_consensus.cc:2391] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 2: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
W20250831 07:58:54.437912 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 260968us
I20250831 07:58:54.571303 871 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:54.620096 667 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:54.577670 862 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:54.580741 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:54.585377 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:54.584527 31725 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:54.589299 594 raft_consensus.cc:2762] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250831 07:58:54.591151 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:54.637506 32467 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:54.609150 803 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.641860 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:54.586877 31806 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:54.597334 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:54.643750 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1459) took 1217 ms. Trace:
I20250831 07:58:54.549863 745 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:54.605985 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:54.609680 31726 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:54.607970 753 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.608142 735 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.439711 854 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:54.611449 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:54.614246 869 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:54.572829 790 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:54.615700 31800 raft_consensus.cc:1238] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:54.622035 31880 rpcz_store.cc:276] 0831 07:58:53.130656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.130811 (+ 155us) service_pool.cc:225] Handling call
0831 07:58:53.131161 (+ 350us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:53.131488 (+ 327us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:53.131516 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:58:53.138168 (+ 6652us) log.cc:844] Serialized 26 byte log entry
0831 07:58:53.219707 (+ 81539us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:58:53.219745 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:53.219810 (+ 65us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:58:54.545765 (+1325955us) raft_consensus.cc:1683] finished
0831 07:58:54.545798 (+ 33us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:54.546013 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2181,"raft.queue_time_us":184,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":9670,"thread_start_us":57725,"threads_started":1,"wal-append.queue_time_us":298220}
I20250831 07:58:54.623301 853 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.573871 31809 raft_consensus.cc:604] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Received request to transfer leadership to TS 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:54.596647 642 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:54.623613 747 consensus_meta.cc:287] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.397s user 0.012s sys 0.000s
I20250831 07:58:54.471530 677 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:54.621917 779 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:54.641031 667 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:54.641304 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:54.591034 851 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:54.642596 31734 raft_consensus.cc:2466] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:54.612386 575 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:54.574329 760 consensus_queue.cc:260] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:54.638495 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:58:54.645953 773 consensus_meta.cc:287] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.521s user 0.008s sys 0.002s
I20250831 07:58:54.607609 738 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.601581 827 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.647454 31849 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:54.621752 865 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:54.656147 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 442427us
I20250831 07:58:54.631899 871 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:54.621843 730 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.644093 31733 rpcz_store.cc:276] 0831 07:58:53.426482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:54.597031 (+1170549us) service_pool.cc:225] Handling call
0831 07:58:54.643696 (+ 46665us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:54.567796 867 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:54.723713 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:54.969283 31809 raft_consensus.cc:612] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:58:54.822038 31728 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:54.969944 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:54.970937 31699 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:54.772831 31804 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:54.880064 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:54.982496 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1434) took 2113 ms. Trace:
W20250831 07:58:54.968833 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 321434us
W20250831 07:58:54.831168 571 consensus_meta.cc:287] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.349s user 0.009s sys 0.004s
I20250831 07:58:54.986480 31800 rpcz_store.cc:276] 0831 07:58:52.868609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:54.590867 (+1722258us) service_pool.cc:225] Handling call
0831 07:58:54.615468 (+ 24601us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:54.982218 (+366750us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:54.982440 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:54.985975 773 raft_consensus.cc:2747] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:54.810876 782 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.885025 833 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:54.988154 571 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:54.975893 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1461) took 1123 ms. Trace:
I20250831 07:58:54.655138 771 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.745221 32466 raft_consensus.cc:2391] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:54.778311 31848 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:58:54.863372 743 consensus_meta.cc:287] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.712s user 0.001s sys 0.010s
I20250831 07:58:54.746459 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1040) took 1079 ms. Trace:
W20250831 07:58:54.825924 31732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 986) took 1603 ms (client timeout 1731 ms). Trace:
I20250831 07:58:54.643373 627 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:58:54.822311 804 consensus_meta.cc:287] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 1.252s user 0.002s sys 0.010s
I20250831 07:58:54.802727 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:54.774624 31805 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
W20250831 07:58:54.825040 31727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 981) took 1604 ms (client timeout 2079 ms). Trace:
I20250831 07:58:54.864540 683 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:58:54.746673 31726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 978) took 1437 ms (client timeout 1578 ms). Trace:
I20250831 07:58:54.658071 31731 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:54.746527 31881 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:58:54.654760 781 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:54.988819 571 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.656916 767 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:55.002326 804 raft_consensus.cc:2747] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:54.968261 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1464) took 1102 ms. Trace:
I20250831 07:58:54.971639 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:54.821266 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:54.992132 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 985) took 1770 ms. Trace:
W20250831 07:58:55.025908 31700 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:57688 dropped due to backpressure: service queue is full with 50 items
W20250831 07:58:54.773751 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:54.951737 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:54.815330 703 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.027428 31734 rpcz_store.cc:276] 0831 07:58:53.865922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:54.635370 (+769448us) service_pool.cc:225] Handling call
0831 07:58:54.968184 (+332814us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:54.746564 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:55.002080 771 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:58:54.990996 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 212823us
I20250831 07:58:54.995164 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:54.877846 764 consensus_meta.cc:287] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.525s user 0.009s sys 0.003s
W20250831 07:58:54.663934 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:54.894275 848 log.cc:927] Time spent T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 1.435s user 0.015s sys 0.005s
I20250831 07:58:54.633445 784 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:58:54.997763 31732 rpcz_store.cc:269] 0831 07:58:53.222047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.459464 (+237417us) service_pool.cc:225] Handling call
0831 07:58:54.825722 (+1366258us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b5400119ed4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:54.825843 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39161728}
I20250831 07:58:54.997207 31883 rpcz_store.cc:276] 0831 07:58:53.667094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.669289 (+ 2195us) service_pool.cc:225] Handling call
0831 07:58:54.746188 (+1076899us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x7b54002986d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:54.746396 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1387990784}
I20250831 07:58:54.652076 725 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:54.641531 866 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.999495 31730 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
W20250831 07:58:54.972003 672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.008s sys 0.009s
I20250831 07:58:54.999687 31808 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
W20250831 07:58:54.747601 768 consensus_meta.cc:287] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 1.260s user 0.012s sys 0.000s
W20250831 07:58:54.816447 766 consensus_meta.cc:287] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.956s user 0.010s sys 0.002s
I20250831 07:58:54.810055 769 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:55.015460 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:55.038501 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.025252 873 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:54.989913 740 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:55.015307 31878 raft_consensus.cc:1238] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 07:58:54.982343 719 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:54.767179 739 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:54.668323 753 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:58:55.037259 672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s user 0.009s sys 0.009s
I20250831 07:58:54.882275 859 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.775811 32463 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47886 (request call id 267) took 1544 ms. Trace:
I20250831 07:58:55.018374 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:54.664660 857 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:54.989382 32470 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
W20250831 07:58:55.002341 31727 rpcz_store.cc:269] 0831 07:58:53.220677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.220781 (+ 104us) service_pool.cc:225] Handling call
0831 07:58:54.824884 (+1604103us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b54001dfe54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:54.824984 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1144765184}
I20250831 07:58:54.994378 505 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:54.656469 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:54.996378 824 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:58:55.057157 31725 raft_consensus.cc:2391] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 2: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
W20250831 07:58:54.987885 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:54.746584 600 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.002666 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1338) took 2679 ms. Trace:
I20250831 07:58:55.021735 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:55.004024 31775 connection.cc:232] RPC call timeout handler was delayed by -1.2407616719999623s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15780000000000002s for a response
I20250831 07:58:54.991932 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:54.992357 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37044 (request call id 8) took 1764 ms. Trace:
I20250831 07:58:55.028091 31804 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:55.018491 854 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:54.991109 747 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:55.023056 860 consensus_meta.cc:287] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.557s user 0.011s sys 0.000s
I20250831 07:58:55.031422 32467 raft_consensus.cc:2366] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3.
I20250831 07:58:54.991544 575 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:54.994660 642 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.026839 31728 rpcz_store.cc:276] 0831 07:58:53.221742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.221881 (+ 139us) service_pool.cc:225] Handling call
0831 07:58:54.821667 (+1599786us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b5800000f98. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:58:54.821853 (+ 186us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:54.983992 (+162139us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:54.992071 (+ 8079us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5844736}
I20250831 07:58:54.997324 779 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.089485 779 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:55.030139 32469 raft_consensus.cc:2391] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:55.091303 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47886 (request call id 268) took 1863 ms. Trace:
I20250831 07:58:55.091492 31698 leader_election.cc:400] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3.
I20250831 07:58:55.091930 31698 leader_election.cc:403] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250831 07:58:55.037151 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:55.091773 32469 rpcz_store.cc:276] 0831 07:58:53.228045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.233581 (+ 5536us) service_pool.cc:225] Handling call
0831 07:58:54.370593 (+1137012us) spinlock_profiling.cc:243] Waited 45.6 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:58:55.091107 (+720514us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b540062f054. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:55.091248 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1152044032}
I20250831 07:58:55.093237 575 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:55.082428 31874 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:55.049907 873 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.094388 814 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:55.094612 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.095296 31874 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:55.095659 873 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:55.034610 771 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.096160 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.096693 31800 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:55.097785 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.098131 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:55.050021 784 consensus_queue.cc:1046] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20250831 07:58:55.098970 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:55.099530 575 raft_consensus.cc:3053] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:55.099944 575 raft_consensus.cc:738] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:55.100626 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1045) took 1257 ms. Trace:
I20250831 07:58:55.100739 575 consensus_queue.cc:260] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:55.101249 771 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:55.035496 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:55.028751 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:58:55.026996 31731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 64) took 1748 ms (client timeout 2163 ms). Trace:
I20250831 07:58:55.002589 869 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:55.102072 31731 rpcz_store.cc:269] 0831 07:58:53.277539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:54.074656 (+797117us) service_pool.cc:225] Handling call
0831 07:58:55.025737 (+951081us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:55.040771 769 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:55.103457 869 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:54.996397 544 raft_consensus.cc:2747] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250831 07:58:55.020380 31726 rpcz_store.cc:269] 0831 07:58:53.220088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.225184 (+ 5096us) service_pool.cc:225] Handling call
0831 07:58:54.653906 (+1428722us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:55.104773 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:55.105010 769 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.101004 31879 rpcz_store.cc:276] 0831 07:58:53.842679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.842801 (+ 122us) service_pool.cc:225] Handling call
0831 07:58:55.100462 (+1257661us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b5400299fd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:55.100567 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":794986880}
I20250831 07:58:55.039096 31883 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:55.103552 734 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:55.105924 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1050) took 1069 ms. Trace:
I20250831 07:58:55.106389 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.106312 31875 rpcz_store.cc:276] 0831 07:58:54.036556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:54.352611 (+316055us) service_pool.cc:225] Handling call
0831 07:58:54.372816 (+ 20205us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:54.572592 (+199776us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:54.572621 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:54.572665 (+ 44us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:58:54.572696 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:55.105746 (+533050us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b5400298bd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:55.105776 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:55.105862 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":458963712}
I20250831 07:58:55.107062 31883 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:55.107309 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.107910 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.108462 31883 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
W20250831 07:58:55.109148 31807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43428 (request call id 79) took 1896 ms (client timeout 1704 ms). Trace:
I20250831 07:58:55.109195 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:55.109452 31807 rpcz_store.cc:269] 0831 07:58:53.212789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.447652 (+234863us) service_pool.cc:225] Handling call
0831 07:58:55.109024 (+1661372us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54000e97d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:55.109098 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":684599552}
I20250831 07:58:55.109802 31883 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:55.110219 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:55.110565 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.111014 830 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:55.044670 31802 raft_consensus.cc:1273] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:55.050603 32464 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:55.050642 753 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.051482 719 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.040621 740 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:55.034399 790 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:55.056422 31773 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:55.037091 704 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:55.025851 767 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.058579 851 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:55.038323 725 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:55.058733 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 67) took 1780 ms. Trace:
I20250831 07:58:55.070549 505 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.129417 31725 rpcz_store.cc:276] 0831 07:58:53.278523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:54.994832 (+1716309us) service_pool.cc:225] Handling call
0831 07:58:55.058680 (+ 63848us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:55.065461 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:55.058473 677 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.125929 790 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.036900 778 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:55.074120 863 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:55.076220 860 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:55.138023 863 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.138793 505 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:58:55.138963 32471 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38362 (request call id 257) took 2093 ms (client timeout 2168 ms). Trace:
I20250831 07:58:55.139472 31876 raft_consensus.cc:1238] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:55.076447 768 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:58:55.139489 32471 rpcz_store.cc:269] 0831 07:58:53.045310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.045503 (+ 193us) service_pool.cc:225] Handling call
0831 07:58:55.138787 (+2093284us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54005bfc54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:55.138903 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3330985728}
I20250831 07:58:55.140363 768 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:55.140889 863 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:58:55.139106 31805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1442) took 1388 ms (client timeout 1771 ms). Trace:
W20250831 07:58:55.077322 31877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1043) took 1409 ms (client timeout 1853 ms). Trace:
I20250831 07:58:55.077843 747 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:55.078440 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 995) took 1502 ms. Trace:
W20250831 07:58:55.082221 31804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43428 (request call id 71) took 3140 ms (client timeout 2163 ms). Trace:
I20250831 07:58:55.149509 31730 rpcz_store.cc:276] 0831 07:58:53.576179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:54.429780 (+853601us) service_pool.cc:225] Handling call
0831 07:58:55.078299 (+648519us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b540013fe54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:55.078388 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":991442176}
I20250831 07:58:55.086562 686 raft_consensus.cc:3053] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:55.087666 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1444) took 1336 ms. Trace:
I20250831 07:58:55.072608 31803 rpcz_store.cc:276] 0831 07:58:52.323185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.042036 (+718851us) service_pool.cc:225] Handling call
0831 07:58:53.104596 (+ 62560us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:53.163105 (+ 58509us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:53.164144 (+ 1039us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:53.168180 (+ 4036us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:53.171646 (+ 3466us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:54.990682 (+1819036us) spinlock_profiling.cc:243] Waited 48.2 ms on lock 0x7b540015b654. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:54.994591 (+ 3909us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:55.000384 (+ 5793us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110773632}
I20250831 07:58:55.090083 642 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:55.105980 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:54.991076 31733 rpcz_store.cc:276] 0831 07:58:53.852680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:54.969691 (+1117011us) service_pool.cc:225] Handling call
0831 07:58:54.975805 (+ 6114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:54.997450 865 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:55.114046 32435 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:55.114910 31883 raft_consensus.cc:1273] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:58:55.057150 824 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:55.073721 730 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:55.187633 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:55.116807 740 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:55.186869 31806 rpcz_store.cc:276] 0831 07:58:53.752646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:54.178475 (+425829us) service_pool.cc:225] Handling call
0831 07:58:55.087520 (+909045us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54003002d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:55.087608 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":459381120}
I20250831 07:58:55.112687 855 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:55.119215 719 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.114542 792 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:55.193648 792 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.082612 688 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:55.121061 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1382) took 2021 ms. Trace:
W20250831 07:58:55.122138 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 65800.2us
I20250831 07:58:55.122561 871 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:55.101984 575 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:55.131774 851 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:55.203941 31878 rpcz_store.cc:276] 0831 07:58:53.099508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.099641 (+ 133us) service_pool.cc:225] Handling call
0831 07:58:53.099986 (+ 345us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:55.040010 (+1940024us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:55.120826 (+ 80816us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b540020c6d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:55.121009 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4374024448}
I20250831 07:58:55.027122 31700 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:57688 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 74)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 76)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 989)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1477)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1471)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1467)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1478)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1470)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 75)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 77)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1001)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 999)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1002)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1005)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1003)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 996)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 79)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1004)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1480)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 988)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 63)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 69)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 70)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 71)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1441)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1442)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1444)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1445)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1446)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1449)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1450)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1451)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1452)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1454)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1460)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1462)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 78)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 997)
I20250831 07:58:55.124329 704 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:55.112732 814 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3.
I20250831 07:58:55.099509 31874 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:55.207841 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.073951 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:55.142235 31805 rpcz_store.cc:269] 0831 07:58:53.750792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:54.133826 (+383034us) service_pool.cc:225] Handling call
0831 07:58:55.138924 (+1005098us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b5400358c54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:55.139045 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1246536576}
I20250831 07:58:55.210489 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:55.142843 31807 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
W20250831 07:58:55.142577 31877 rpcz_store.cc:269] 0831 07:58:53.668259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.670239 (+ 1980us) service_pool.cc:225] Handling call
0831 07:58:55.077176 (+1406937us) spinlock_profiling.cc:243] Waited 34.9 ms on lock 0x7b5400058554. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:55.077271 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":80200448}
I20250831 07:58:55.126593 771 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250831 07:58:55.149587 31804 rpcz_store.cc:269] 0831 07:58:51.934410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.028703 (+ 94293us) service_pool.cc:225] Handling call
0831 07:58:52.913610 (+884907us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b1800024720. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce8038c66 00007fdce7f89d1e 00007fdce7ff3f5e 00007fdce7ff579e 00007fdce800122c 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc
0831 07:58:53.560825 (+647215us) spinlock_profiling.cc:243] Waited 25.2 ms on lock 0x7b580010dfb0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7f78003 00007fdce7fdb577 00007fdce7ff3fa7 00007fdce7ff579e 00007fdce800122c 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc
0831 07:58:54.772457 (+1211632us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b580010e2b0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7f78003 00007fdce7fdb577 00007fdce7ff3fa7 00007fdce7ff579e 00007fdce800122c 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc
0831 07:58:55.074164 (+301707us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2304388352}
I20250831 07:58:55.073338 31809 rpcz_store.cc:276] 0831 07:58:53.205156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:54.544238 (+1339082us) service_pool.cc:225] Handling call
0831 07:58:54.969796 (+425558us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:55.188794 686 raft_consensus.cc:738] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:55.180686 31879 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:55.148764 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1332) took 3464 ms. Trace:
I20250831 07:58:55.187105 31882 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:55.147426 31630 catalog_manager.cc:5582] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c reported cstate change: config changed from index 2 to 3, 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:55.119196 753 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:58:55.220198 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:55.222638 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:55.220026 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:55.217001 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.224498 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:58:55.189549 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 75590.4us
I20250831 07:58:55.142194 768 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.227265 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:55.227702 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 99924.2us
I20250831 07:58:55.228312 31879 raft_consensus.cc:2391] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:55.143687 778 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:55.191125 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.192137 836 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:55.192297 867 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.202899 31800 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:55.230718 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:55.212777 771 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:55.202643 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.144904 571 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:55.055394 32463 rpcz_store.cc:276] 0831 07:58:53.222766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.226484 (+ 3718us) service_pool.cc:225] Handling call
0831 07:58:54.599314 (+1372830us) spinlock_profiling.cc:243] Waited 5.81 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:58:54.631526 (+ 32212us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:54.640258 (+ 8732us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:54.640314 (+ 56us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:54.646230 (+ 5916us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:58:54.651955 (+ 5725us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:54.656754 (+ 4799us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:54.767540 (+110786us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13365120}
I20250831 07:58:55.211380 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.211874 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.188702 824 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:55.212837 31805 raft_consensus.cc:2391] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:55.213593 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:55.209573 688 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:55.216707 31802 rpcz_store.cc:276] 0831 07:58:51.684598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:51.809321 (+124723us) service_pool.cc:225] Handling call
0831 07:58:51.838280 (+ 28959us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:55.111967 (+3273687us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:55.148464 (+ 36497us) spinlock_profiling.cc:243] Waited 3.21 s on lock 0x7b54001da454. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:55.148689 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7373214464}
I20250831 07:58:55.218447 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:55.239183 31772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39068 (request call id 495) took 3547 ms. Trace:
W20250831 07:58:55.226223 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:55.206828 855 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:55.195279 677 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:55.279589 677 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:55.280016 677 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:55.263633 686 consensus_queue.cc:260] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:55.281435 686 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:55.284284 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:55.263795 747 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:58:55.205973 31773 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.618s (SENT) [suppressed 5 similar messages]
I20250831 07:58:55.284938 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:55.285276 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 79351.5us
I20250831 07:58:55.238314 836 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.286187 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:55.286980 31726 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:55.231355 32468 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 1.
I20250831 07:58:55.287840 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:55.289070 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:55.290102 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:55.287612 31773 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:55.236011 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.232137 865 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.233963 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:55.237578 31877 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:55.291778 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.238898 31878 raft_consensus.cc:2391] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:55.291596 31804 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:55.292439 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.240077 31809 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:55.215504 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.188292 31733 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:55.293450 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.247476 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:55.293969 31882 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
W20250831 07:58:55.256798 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:55.235623 771 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:55.263099 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:55.237020 730 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:55.236011 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:55.263993 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.264853 767 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.268770 711 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:55.266487 31700 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:57688 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1470)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 75)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 77)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1001)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 999)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1006)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1002)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1005)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1003)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 996)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1011)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1016)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1015)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 79)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1004)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1013)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1014)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1012)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1010)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1480)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1007)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1008)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 988)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 63)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 69)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 70)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 71)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1441)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1442)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1444)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1445)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1446)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1449)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1450)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1451)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1452)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1454)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1460)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1462)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 78)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 997)
I20250831 07:58:55.294303 31802 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:55.345170 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.345850 31883 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:55.346350 824 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.302523 31731 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
W20250831 07:58:55.315145 32464 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38362 (request call id 260) took 1858 ms (client timeout 1618 ms). Trace:
I20250831 07:58:55.356005 31881 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
W20250831 07:58:55.306983 31808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1435) took 2438 ms (client timeout 3211 ms). Trace:
I20250831 07:58:55.312309 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:55.232395 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:55.306453 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:55.358153 31727 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:55.290670 31726 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:55.237968 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:55.365197 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1469) took 1181 ms. Trace:
W20250831 07:58:55.249662 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 135381us
I20250831 07:58:55.365656 31730 rpcz_store.cc:276] 0831 07:58:54.183524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.306168 (+1122644us) service_pool.cc:225] Handling call
0831 07:58:55.365136 (+ 58968us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:55.348738 32464 rpcz_store.cc:269] 0831 07:58:53.457149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.460200 (+ 3051us) service_pool.cc:225] Handling call
0831 07:58:55.314964 (+1854764us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54003ba954. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:55.315069 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3059192064}
W20250831 07:58:55.357117 31808 rpcz_store.cc:269] 0831 07:58:52.868951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:52.872107 (+ 3156us) service_pool.cc:225] Handling call
0831 07:58:55.306779 (+2434672us) spinlock_profiling.cc:243] Waited 2.1 s on lock 0x7b54000cf8d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:55.306916 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4825863936}
I20250831 07:58:55.302337 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:55.311699 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:55.310222 31628 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43950 (request call id 119) took 6427 ms. Trace:
I20250831 07:58:55.356504 730 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:56.529155 31700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 997) took 1525 ms. Trace:
I20250831 07:58:55.278079 760 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:55.284940 747 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:56.529557 31700 rpcz_store.cc:276] 0831 07:58:55.003761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:56.529095 (+1525334us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:56.530371 31700 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:57688 dropped due to backpressure: service queue is full with 50 items [suppressed 2 similar messages]
I20250831 07:58:55.318296 772 raft_consensus.cc:3053] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:56.531373 772 raft_consensus.cc:738] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:55.304229 725 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.306702 851 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:56.532635 772 consensus_queue.cc:260] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:56.530647 31700 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:57688 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1001)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 999)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1006)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1002)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1005)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1020)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1003)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1019)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 996)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1021)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1011)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1016)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1017)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1015)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1018)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 79)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1004)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1013)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1014)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1012)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1010)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1022)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1480)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1007)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1008)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1023)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 988)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 63)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 69)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 70)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 71)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1441)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1442)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1444)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1445)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1446)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1449)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1450)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1451)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1452)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1454)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1460)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1462)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 78)
I20250831 07:58:56.533694 772 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:55.293207 836 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:56.529604 730 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:55.350632 704 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.353392 792 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.342300 855 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.369239 31698 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:58:57.270113 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 1.90087e+06us
I20250831 07:58:55.359715 31725 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
W20250831 07:58:57.299638 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 76) took 2733 ms (client timeout 1662 ms). Trace:
W20250831 07:58:57.300056 31725 rpcz_store.cc:269] 0831 07:58:54.566151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.215945 (+649794us) service_pool.cc:225] Handling call
0831 07:58:57.299570 (+2083625us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.300889 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1001) took 2295 ms (client timeout 1682 ms). Trace:
W20250831 07:58:57.301147 31725 rpcz_store.cc:269] 0831 07:58:55.005183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.300696 (+2295513us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.300802 (+ 106us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.301709 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 999) took 2297 ms (client timeout 1691 ms). Trace:
W20250831 07:58:57.301921 31725 rpcz_store.cc:269] 0831 07:58:55.004592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.301600 (+2297008us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.301666 (+ 66us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.302626 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1473) took 2838 ms (client timeout 2249 ms). Trace:
W20250831 07:58:57.302860 31725 rpcz_store.cc:269] 0831 07:58:54.464784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.302521 (+2837737us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.302592 (+ 71us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.303350 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1479) took 2656 ms (client timeout 2158 ms). Trace:
I20250831 07:58:55.375147 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1448) took 2035 ms. Trace:
I20250831 07:58:55.365795 31732 raft_consensus.cc:2391] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:55.366658 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:55.348372 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1453) took 2009 ms. Trace:
I20250831 07:58:55.359037 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:55.294327 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:55.905387 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:57.308161 31727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 75) took 2921 ms (client timeout 2163 ms). Trace:
W20250831 07:58:56.302284 31630 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.989s user 0.002s sys 0.036s
I20250831 07:58:55.370507 31772 rpcz_store.cc:276] 0831 07:58:51.692292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.239142 (+3546850us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:55.368439 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 80933.1us
W20250831 07:58:55.689361 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:56.316743 31734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 74) took 2083 ms (client timeout 1971 ms). Trace:
I20250831 07:58:55.363867 31803 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:55.358472 31805 raft_consensus.cc:2374] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2. Re-sending same reply.
W20250831 07:58:57.315572 31805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43428 (request call id 83) took 2646 ms (client timeout 2163 ms). Trace:
W20250831 07:58:57.315852 31805 rpcz_store.cc:269] 0831 07:58:54.669203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.250548 (+581345us) service_pool.cc:225] Handling call
0831 07:58:57.315521 (+2064973us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.316419 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:57.316576 31805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1450) took 2192 ms (client timeout 1886 ms). Trace:
I20250831 07:58:57.316596 902 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:55.241367 31806 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:55.430255 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:57.317080 31805 rpcz_store.cc:269] 0831 07:58:55.124583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.316441 (+2191858us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.316517 (+ 76us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 07:58:57.318572 31808 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:55.372056 31800 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:55.416368 31802 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:58:55.372545 31804 raft_consensus.cc:2391] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:55.342244 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:55.469030 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 109913us
W20250831 07:58:55.570277 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:55.465195 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:57.323079 31849 connection.cc:232] RPC call timeout handler was delayed by -1.7390209699999559s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2189s for a response
W20250831 07:58:57.323314 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:57.323603 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:57.324018 31849 connection.cc:232] RPC call timeout handler was delayed by -1.6982443059999923s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3047s for a response
I20250831 07:58:55.295928 31877 raft_consensus.cc:2408] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250831 07:58:57.326848 575 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:58:57.326992 31877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1414) took 2056 ms (client timeout 2004 ms). Trace:
W20250831 07:58:57.327586 31877 rpcz_store.cc:269] 0831 07:58:55.270630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.293247 (+ 22617us) service_pool.cc:225] Handling call
0831 07:58:57.326904 (+2033657us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:55.305503 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.341751 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:56.309303 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1416) took 1038 ms. Trace:
I20250831 07:58:57.330396 31881 rpcz_store.cc:276] 0831 07:58:55.271202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.291540 (+ 20338us) service_pool.cc:225] Handling call
0831 07:58:56.309079 (+1017539us) spinlock_profiling.cc:243] Waited 63.6 ms on lock 0x7b5400246154. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:56.309229 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":146191616}
I20250831 07:58:57.331593 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:57.332320 31881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1424) took 1883 ms (client timeout 2099 ms). Trace:
W20250831 07:58:57.332657 31881 rpcz_store.cc:269] 0831 07:58:55.449068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.331366 (+1882298us) service_pool.cc:225] Handling call
0831 07:58:57.332267 (+ 901us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:55.318413 31874 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:58:57.333678 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:57.334268 31874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 89) took 2255 ms (client timeout 2049 ms). Trace:
I20250831 07:58:57.334596 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1426) took 1880 ms. Trace:
W20250831 07:58:57.334802 31874 rpcz_store.cc:269] 0831 07:58:55.079223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.098730 (+ 19507us) service_pool.cc:225] Handling call
0831 07:58:57.334153 (+2235423us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.335157 31881 rpcz_store.cc:276] 0831 07:58:55.454402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.333390 (+1878988us) service_pool.cc:225] Handling call
0831 07:58:57.334531 (+ 1141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.337020 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1425) took 1886 ms. Trace:
I20250831 07:58:57.337538 31874 rpcz_store.cc:276] 0831 07:58:55.450932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.335905 (+1884973us) service_pool.cc:225] Handling call
0831 07:58:57.336301 (+ 396us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.336550 (+ 249us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.336592 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.336659 (+ 67us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:57.336713 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.336802 (+ 89us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.336946 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:55.481555 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:55.434772 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:55.509230 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:55.568092 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:56.268486 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 63782.3us
W20250831 07:58:57.343380 31848 connection.cc:232] RPC call timeout handler was delayed by -1.7865521390000367s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2189s for a response
I20250831 07:58:57.343538 904 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:56.534296 31700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 78) took 1697 ms. Trace:
I20250831 07:58:56.538744 890 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:57.343911 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:55.477101 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:57.344305 890 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:57.344477 904 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:57.346489 907 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:57.346840 904 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:55.472743 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:56.528182 31628 rpcz_store.cc:276] 0831 07:58:48.886402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:48.916238 (+ 29836us) service_pool.cc:225] Handling call
0831 07:58:49.237533 (+321295us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b54000af354. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f81cd0 00007fdce8005eea 00007fdcedd2417c 00007fdcede1cb5e 00007fdcec365484 00007fdcec365412 00007fdcec365341 00007fdcec3652bc 00007fdcec364572 00007fdce63fcccc 00007fdce63fc106
0831 07:58:55.310138 (+6072605us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:58:51.518964 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:58:51.656863 (+137899us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:58:51.661458 (+ 4595us) write_op.cc:435] Acquired schema lock
0831 07:58:51.661490 (+ 32us) tablet.cc:660] Decoding operations
0831 07:58:51.772839 (+111349us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:58:51.776272 (+ 3433us) write_op.cc:641] Partition lock acquired for write op
0831 07:58:51.782299 (+ 6027us) tablet.cc:683] Acquiring locks for 4 operations
0831 07:58:51.867850 (+ 85551us) tablet.cc:699] Row locks acquired
0831 07:58:51.867875 (+ 25us) write_op.cc:260] PREPARE: finished
0831 07:58:51.884254 (+ 16379us) write_op.cc:270] Start()
0831 07:58:51.896663 (+ 12409us) write_op.cc:276] Timestamp: P: 1756627131879311 usec, L: 0
0831 07:58:51.896682 (+ 19us) op_driver.cc:348] REPLICATION: starting
0831 07:58:51.911051 (+ 14369us) log.cc:844] Serialized 1736 byte log entry
0831 07:58:53.052982 (+1141931us) spinlock_profiling.cc:243] Waited 3.25 ms on lock 0x7b180033d7e0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b792 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:58:53.287144 (+234162us) op_driver.cc:464] REPLICATION: finished
0831 07:58:53.621491 (+334347us) write_op.cc:301] APPLY: starting
0831 07:58:53.685345 (+ 63854us) tablet.cc:1365] starting BulkCheckPresence
0831 07:58:53.735325 (+ 49980us) tablet.cc:1368] finished BulkCheckPresence
0831 07:58:53.741550 (+ 6225us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:58:53.838291 (+ 96741us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:58:53.838469 (+ 178us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0831 07:58:53.838511 (+ 42us) write_op.cc:312] APPLY: finished
0831 07:58:53.838727 (+ 216us) log.cc:844] Serialized 41 byte log entry
0831 07:58:53.838889 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:53.838988 (+ 99us) write_op.cc:454] Released schema lock
0831 07:58:53.839040 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0831 07:58:53.839392 (+ 352us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:53.839420 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":99110,"reported_tablets":6,"spinlock_wait_cycles":581873920,"child_traces":[["op",{"apply.queue_time_us":287284,"apply.run_cpu_time_us":3200,"apply.run_wall_time_us":228377,"num_ops":4,"prepare.queue_time_us":276019,"prepare.run_cpu_time_us":14959,"prepare.run_wall_time_us":511219,"raft.queue_time_us":147777,"raft.run_cpu_time_us":31922,"raft.run_wall_time_us":273022,"replication_time_us":1336824,"spinlock_wait_cycles":7479168,"thread_start_us":419532,"threads_started":4,"wal-append.queue_time_us":100888,"wal-append.run_cpu_time_us":27965,"wal-append.run_wall_time_us":1809073}]]}
I20250831 07:58:57.349067 32471 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:55.485128 677 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:55.456792 688 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.370513 711 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.354971 32470 raft_consensus.cc:2366] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3.
I20250831 07:58:55.371439 735 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:55.301903 740 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.362330 740 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:56.537225 747 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:57.364027 31880 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:57.364892 711 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:57.365408 31880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1058) took 2721 ms (client timeout 1964 ms). Trace:
W20250831 07:58:57.365960 31880 rpcz_store.cc:269] 0831 07:58:54.644277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.073697 (+429420us) service_pool.cc:225] Handling call
0831 07:58:57.365197 (+2291500us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b54002bcd54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:57.365330 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4953332992}
I20250831 07:58:55.571264 754 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:57.368322 754 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:57.370952 754 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:58:55.904043 771 consensus_meta.cc:287] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.531s user 0.005s sys 0.005s
I20250831 07:58:57.373917 771 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:56.518675 778 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:55.375218 790 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:55.468725 824 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:58:57.016842 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 203498us
W20250831 07:58:57.195605 31729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 987) took 3973 ms (client timeout 1965 ms). Trace:
W20250831 07:58:57.379680 31729 rpcz_store.cc:269] 0831 07:58:53.222244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:53.686880 (+464636us) service_pool.cc:225] Handling call
0831 07:58:57.195331 (+3508451us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b540038d1d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:57.195491 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1323769216}
W20250831 07:58:57.381192 31729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1006) took 2167 ms (client timeout 1723 ms). Trace:
I20250831 07:58:55.027649 31699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:57688 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 74)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 76)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 989)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1477)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1471)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1467)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1478)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1470)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 75)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 77)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1001)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 999)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1002)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1005)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1003)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 996)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 79)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1004)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1480)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 988)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 63)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 69)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 70)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 71)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1441)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1442)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1444)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1445)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1446)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1449)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1450)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1451)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1452)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1454)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1460)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1462)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 78)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 997)
W20250831 07:58:57.381608 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:57.382705 31775 leader_election.cc:336] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.578s (SENT)
I20250831 07:58:57.383203 31775 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:57.385995 771 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:57.386799 778 raft_consensus.cc:2747] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250831 07:58:57.381637 31729 rpcz_store.cc:269] 0831 07:58:55.214261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.380925 (+2166664us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.381095 (+ 170us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.272521 31698 connection.cc:232] RPC call timeout handler was delayed by -1.293909675000009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
I20250831 07:58:57.271785 898 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:57.390229 898 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:57.390714 898 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:58:57.303558 31725 rpcz_store.cc:269] 0831 07:58:54.646776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.303258 (+2656482us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.303321 (+ 63us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 07:58:57.305084 31726 rpcz_store.cc:276] 0831 07:58:53.337480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.289870 (+1952390us) service_pool.cc:225] Handling call
0831 07:58:55.373115 (+ 83245us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.306347 31732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 989) took 4083 ms (client timeout 3031 ms). Trace:
W20250831 07:58:57.392794 31732 rpcz_store.cc:269] 0831 07:58:53.222592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.237793 (+2015201us) service_pool.cc:225] Handling call
0831 07:58:57.306292 (+2068499us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.306995 31730 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
W20250831 07:58:57.394042 31730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 77) took 2783 ms (client timeout 2066 ms). Trace:
W20250831 07:58:57.394521 31730 rpcz_store.cc:269] 0831 07:58:54.610559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.366425 (+755866us) service_pool.cc:225] Handling call
0831 07:58:57.393956 (+2027531us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.307487 31731 rpcz_store.cc:276] 0831 07:58:53.338863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.284669 (+1945806us) service_pool.cc:225] Handling call
0831 07:58:55.348314 (+ 63645us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:56.295892 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:57.308614 31733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1476) took 2792 ms (client timeout 2033 ms). Trace:
W20250831 07:58:57.395869 31698 connection.cc:232] RPC call timeout handler was delayed by -1.2750096749999784s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
W20250831 07:58:57.395702 31733 rpcz_store.cc:269] 0831 07:58:54.516258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.294093 (+777835us) service_pool.cc:225] Handling call
0831 07:58:57.308553 (+2014460us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.308558 31727 rpcz_store.cc:269] 0831 07:58:54.386599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.358777 (+972178us) service_pool.cc:225] Handling call
0831 07:58:57.308111 (+1949334us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.308751 31728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1467) took 3126 ms (client timeout 2140 ms). Trace:
W20250831 07:58:57.396761 31728 rpcz_store.cc:269] 0831 07:58:54.182852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.905110 (+1722258us) service_pool.cc:225] Handling call
0831 07:58:57.308688 (+1403578us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.308797 31630 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.996s user 0.003s sys 0.037s
I20250831 07:58:57.310508 31772 service_pool.cc:138] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:39068 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1455)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1355)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1454)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1457)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1456)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43428 (request call id 85)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1458)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1436)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 72)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 73)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 74)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 76)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 77)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 78)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1439)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1441)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1341)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1343)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1445)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 80)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 81)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1353)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1354)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1356)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1361)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1449)
I20250831 07:58:57.396764 704 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:57.312888 899 consensus_queue.cc:1035] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:57.400767 704 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.312919 31773 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:58:57.402631 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 89795.4us
W20250831 07:58:57.313645 31734 rpcz_store.cc:269] 0831 07:58:54.233538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:56.316402 (+2082864us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:56.316623 (+ 221us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.314711 31803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1349) took 2238 ms (client timeout 1634 ms). Trace:
I20250831 07:58:57.314873 900 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:57.317123 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:57.317227 902 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:57.317945 31806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1351) took 2240 ms (client timeout 1667 ms). Trace:
W20250831 07:58:57.318681 31805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1358) took 2239 ms (client timeout 2007 ms). Trace:
W20250831 07:58:57.319072 31808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1364) took 2236 ms (client timeout 1903 ms). Trace:
I20250831 07:58:55.308348 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:57.319654 31800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1447) took 2203 ms (client timeout 1837 ms). Trace:
W20250831 07:58:57.320108 31802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1460) took 2085 ms (client timeout 1500 ms). Trace:
I20250831 07:58:57.406131 31809 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
W20250831 07:58:57.406291 31802 rpcz_store.cc:269] 0831 07:58:55.234734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.240709 (+ 5975us) service_pool.cc:225] Handling call
0831 07:58:57.320060 (+2079351us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.320640 31804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1357) took 2242 ms (client timeout 1852 ms). Trace:
W20250831 07:58:57.407006 31809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1459) took 2177 ms (client timeout 1701 ms). Trace:
W20250831 07:58:57.407218 31804 rpcz_store.cc:269] 0831 07:58:55.078575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.301945 (+223370us) service_pool.cc:225] Handling call
0831 07:58:57.320577 (+2018632us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:55.467274 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:57.404574 31805 rpcz_store.cc:269] 0831 07:58:55.078838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.318539 (+2239701us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.318623 (+ 84us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 07:58:57.329303 31876 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:57.403781 31773 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:58:57.408756 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:57.408818 31876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1418) took 2137 ms (client timeout 2478 ms). Trace:
W20250831 07:58:57.409248 31876 rpcz_store.cc:269] 0831 07:58:55.274200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.305232 (+ 31032us) service_pool.cc:225] Handling call
0831 07:58:57.408763 (+2103531us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.329246 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:57.409722 31773 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:55.306306 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:57.418395 31875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1411) took 2148 ms (client timeout 2075 ms). Trace:
W20250831 07:58:57.418692 31733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1005) took 2412 ms (client timeout 1964 ms). Trace:
W20250831 07:58:57.418890 31875 rpcz_store.cc:269] 0831 07:58:55.269795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.305979 (+ 36184us) service_pool.cc:225] Handling call
0831 07:58:57.418309 (+2112330us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.419127 31733 rpcz_store.cc:269] 0831 07:58:55.006313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.418530 (+2412217us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.418636 (+ 106us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.420143 31733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 996) took 2416 ms (client timeout 2096 ms). Trace:
W20250831 07:58:57.420446 31733 rpcz_store.cc:269] 0831 07:58:55.003404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.420001 (+2416597us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.420096 (+ 95us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.420755 31726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1019) took 2012 ms (client timeout 1661 ms). Trace:
W20250831 07:58:57.421195 31730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1021) took 1988 ms (client timeout 1731 ms). Trace:
W20250831 07:58:57.421828 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1020) took 1994 ms (client timeout 1578 ms). Trace:
W20250831 07:58:57.422199 31725 rpcz_store.cc:269] 0831 07:58:55.427874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.421653 (+1993779us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.421754 (+ 101us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.405972 31800 rpcz_store.cc:269] 0831 07:58:55.116052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.311365 (+195313us) service_pool.cc:225] Handling call
0831 07:58:57.319597 (+2008232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.423077 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:57.423445 31800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1455) took 2226 ms (client timeout 1932 ms). Trace:
W20250831 07:58:57.424296 31727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1016) took 2158 ms (client timeout 1939 ms). Trace:
W20250831 07:58:57.424005 31800 rpcz_store.cc:269] 0831 07:58:55.196594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.423241 (+2226647us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.423375 (+ 134us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.424858 31732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1003) took 2419 ms (client timeout 2044 ms). Trace:
I20250831 07:58:57.405712 902 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:58:56.318186 31882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 88) took 1540 ms (client timeout 1662 ms). Trace:
I20250831 07:58:57.423722 31725 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
W20250831 07:58:57.426795 31882 rpcz_store.cc:269] 0831 07:58:54.777802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.027121 (+249319us) service_pool.cc:225] Handling call
0831 07:58:56.317943 (+1290822us) spinlock_profiling.cc:243] Waited 94.3 ms on lock 0x7b5400057b54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:56.318115 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":217002752}
I20250831 07:58:57.340658 31883 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
W20250831 07:58:57.421651 31730 rpcz_store.cc:269] 0831 07:58:55.432969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.421014 (+1988045us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.421115 (+ 101us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.341686 31849 connection.cc:232] RPC call timeout handler was delayed by -1.3454596310000397s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2008s for a response
W20250831 07:58:57.428196 31734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1002) took 2422 ms (client timeout 1951 ms). Trace:
W20250831 07:58:57.425323 31732 rpcz_store.cc:269] 0831 07:58:55.005743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.424703 (+2418960us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.424802 (+ 99us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.428848 31734 rpcz_store.cc:269] 0831 07:58:55.005473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.427965 (+2422492us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.428118 (+ 153us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 07:58:57.429641 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.429791 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.430207 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.348098 906 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:58:57.424831 31727 rpcz_store.cc:269] 0831 07:58:55.266066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.424131 (+2158065us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.424240 (+ 109us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 07:58:57.341712 905 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:57.431113 906 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.431408 905 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.431660 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.350953 31807 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
W20250831 07:58:57.432834 31727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1014) took 2190 ms (client timeout 2753 ms). Trace:
I20250831 07:58:57.433388 905 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:57.428642 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:57.433655 31807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1461) took 2197 ms (client timeout 1748 ms). Trace:
W20250831 07:58:57.404458 31806 rpcz_store.cc:269] 0831 07:58:55.077145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.204163 (+127018us) service_pool.cc:225] Handling call
0831 07:58:57.317882 (+2113719us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.344069 31700 rpcz_store.cc:276] 0831 07:58:54.837000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:56.534247 (+1697247us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 07:58:57.430406 31729 raft_consensus.cc:2391] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 2: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
W20250831 07:58:57.434470 31806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1365) took 2349 ms (client timeout 2075 ms). Trace:
W20250831 07:58:57.434885 31806 rpcz_store.cc:269] 0831 07:58:55.085282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.434315 (+2349033us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.434417 (+ 102us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 07:58:57.435267 31730 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
W20250831 07:58:57.435734 31806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1454) took 2247 ms (client timeout 2008 ms). Trace:
W20250831 07:58:57.435794 31730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1018) took 2062 ms (client timeout 2167 ms). Trace:
I20250831 07:58:57.434351 31700 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:43230 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1012)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1024)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1010)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1022)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1480)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1007)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1008)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1023)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1486)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1485)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1482)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1484)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 988)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 63)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 69)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 70)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 71)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1441)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1442)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1444)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1445)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1446)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1449)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1450)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1451)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1452)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1454)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1460)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1462)
W20250831 07:58:57.436163 31806 rpcz_store.cc:269] 0831 07:58:55.188362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.435601 (+2247239us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.435687 (+ 86us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.436426 31730 rpcz_store.cc:269] 0831 07:58:55.373759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.428403 (+2054644us) service_pool.cc:225] Handling call
0831 07:58:57.435726 (+ 7323us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.437003 31806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1368) took 2320 ms (client timeout 2083 ms). Trace:
W20250831 07:58:57.437384 31806 rpcz_store.cc:269] 0831 07:58:55.116933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.436863 (+2319930us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.436948 (+ 85us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 07:58:57.437704 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:57.438076 31806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1452) took 2300 ms (client timeout 2140 ms). Trace:
W20250831 07:58:57.438418 31806 rpcz_store.cc:269] 0831 07:58:55.144039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.437926 (+2293887us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.438031 (+ 105us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.439118 31806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1457) took 2227 ms (client timeout 2068 ms). Trace:
I20250831 07:58:57.330098 31878 raft_consensus.cc:2366] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3.
I20250831 07:58:57.356370 31801 raft_consensus.cc:1273] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250831 07:58:57.440152 31878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1415) took 2169 ms (client timeout 1996 ms). Trace:
I20250831 07:58:57.440478 906 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:57.353420 677 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:57.441227 31804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1456) took 2232 ms (client timeout 2108 ms). Trace:
I20250831 07:58:57.441560 677 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:57.366828 890 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:58:57.441601 31804 rpcz_store.cc:269] 0831 07:58:55.208551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.441077 (+2232526us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.441177 (+ 100us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.396260 31698 connection.cc:232] RPC call timeout handler was delayed by -1.2219096750000062s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1627s for a response
W20250831 07:58:57.442502 31804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43428 (request call id 85) took 2352 ms (client timeout 2243 ms). Trace:
W20250831 07:58:57.442855 31804 rpcz_store.cc:269] 0831 07:58:55.113389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.442366 (+2328977us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.442456 (+ 90us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.434037 31807 rpcz_store.cc:269] 0831 07:58:55.235922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.314502 (+ 78580us) service_pool.cc:225] Handling call
0831 07:58:57.433499 (+2118997us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b54002f0654. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:57.433596 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68458880}
W20250831 07:58:57.434685 31729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 79) took 2532 ms (client timeout 2790 ms). Trace:
I20250831 07:58:57.443907 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:57.443802 31729 rpcz_store.cc:269] 0831 07:58:54.901876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.429395 (+2527519us) service_pool.cc:225] Handling call
0831 07:58:57.434619 (+ 5224us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.444362 31731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1017) took 2075 ms (client timeout 1971 ms). Trace:
I20250831 07:58:57.444609 31807 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
W20250831 07:58:57.445106 31807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1367) took 2328 ms (client timeout 2478 ms). Trace:
I20250831 07:58:57.404265 899 consensus_queue.cc:1035] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
W20250831 07:58:57.445662 31807 rpcz_store.cc:269] 0831 07:58:55.116289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.443648 (+2327359us) service_pool.cc:225] Handling call
0831 07:58:57.445041 (+ 1393us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10112}
I20250831 07:58:57.446643 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.445139 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.447249 31807 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:57.447352 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.401052 31772 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:39082 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1455)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1355)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1454)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1457)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1456)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43428 (request call id 85)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1458)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1352)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1464)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1471)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1468)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1463)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1462)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1470)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1465)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1436)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 72)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 73)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 74)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 76)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 77)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 78)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1439)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1441)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1341)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1343)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1445)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 80)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 81)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1353)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1354)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1356)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1361)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1449)
I20250831 07:58:57.327394 575 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:57.408919 31805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1448) took 2291 ms (client timeout 1993 ms). Trace:
W20250831 07:58:57.448148 31805 rpcz_store.cc:269] 0831 07:58:55.117544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.408764 (+2291220us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.408859 (+ 95us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 07:58:57.448268 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.449028 31729 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:57.449115 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.449517 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1022) took 1093 ms. Trace:
I20250831 07:58:57.449831 31805 raft_consensus.cc:2391] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
W20250831 07:58:57.450278 31805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1352) took 2372 ms (client timeout 2676 ms). Trace:
I20250831 07:58:57.450721 575 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:57.410074 31877 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
W20250831 07:58:57.451426 31877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1427) took 1994 ms (client timeout 2050 ms). Trace:
W20250831 07:58:57.451749 31877 rpcz_store.cc:269] 0831 07:58:55.458910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.328921 (+1870011us) service_pool.cc:225] Handling call
0831 07:58:57.451365 (+122444us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.410585 31773 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:58:57.452720 32436 connection.cc:232] RPC call timeout handler was delayed by -1.0111652540000478s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2163s for a response
I20250831 07:58:57.453190 31773 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:57.453907 890 raft_consensus.cc:2802] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:57.454270 890 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:57.411063 778 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:57.454818 890 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:57.440491 31878 rpcz_store.cc:269] 0831 07:58:55.270907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.292957 (+ 22050us) service_pool.cc:225] Handling call
0831 07:58:57.440088 (+2147131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.455109 778 raft_consensus.cc:2802] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:57.455538 778 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:57.455935 778 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:57.404712 31808 rpcz_store.cc:269] 0831 07:58:55.082969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.371869 (+288900us) service_pool.cc:225] Handling call
0831 07:58:57.319012 (+1947143us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.456972 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.457551 31808 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:57.413172 824 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:58:57.444800 31731 rpcz_store.cc:269] 0831 07:58:55.368859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.444195 (+2075336us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.444304 (+ 109us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 07:58:57.458456 711 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:57.412084 911 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:57.459233 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.459656 911 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.384488 31699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:57678 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1002)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1005)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1020)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1003)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1019)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 996)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1021)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1011)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1016)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1017)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1015)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1018)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 79)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1004)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1013)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1014)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1012)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1024)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1010)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1022)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1480)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1007)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1008)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1023)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1486)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1485)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1482)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1484)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 988)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 63)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 65)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 66)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 69)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 70)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 71)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1441)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1442)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1444)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1445)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1446)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1449)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1450)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1451)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1452)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1454)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1460)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1462)
I20250831 07:58:57.425967 902 raft_consensus.cc:2802] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250831 07:58:57.404076 31803 rpcz_store.cc:269] 0831 07:58:55.076661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.204086 (+127425us) service_pool.cc:225] Handling call
0831 07:58:57.314645 (+2110559us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.458413 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:57.461314 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1333) took 5774 ms. Trace:
W20250831 07:58:57.421165 31733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1011) took 2192 ms (client timeout 1939 ms). Trace:
I20250831 07:58:57.339988 31879 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
W20250831 07:58:57.421352 31726 rpcz_store.cc:269] 0831 07:58:55.408412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.420520 (+2012108us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.420675 (+ 155us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.461877 31733 rpcz_store.cc:269] 0831 07:58:55.228527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.421026 (+2192499us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.421118 (+ 92us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.427874 31883 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1423) took 1993 ms (client timeout 2625 ms). Trace:
W20250831 07:58:57.428349 31849 connection.cc:232] RPC call timeout handler was delayed by -1.3276469240000779s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17010000000000003s for a response
I20250831 07:58:57.462672 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.430307 31732 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:57.462973 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:57.463388 31732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1004) took 2457 ms (client timeout 2738 ms). Trace:
I20250831 07:58:57.346956 907 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:57.463696 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1008) took 2248 ms. Trace:
I20250831 07:58:57.461900 902 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:58:57.462698 31883 rpcz_store.cc:269] 0831 07:58:55.434450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.434549 (+ 99us) service_pool.cc:225] Handling call
0831 07:58:57.427800 (+1993251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.464807 902 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:58:57.463862 31732 rpcz_store.cc:269] 0831 07:58:55.006043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.429588 (+2423545us) service_pool.cc:225] Handling call
0831 07:58:57.463335 (+ 33747us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.465490 907 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.465490 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1007) took 2250 ms. Trace:
I20250831 07:58:57.466251 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:57.466444 31726 rpcz_store.cc:276] 0831 07:58:55.214613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.462433 (+2247820us) service_pool.cc:225] Handling call
0831 07:58:57.465413 (+ 2980us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.466833 31732 raft_consensus.cc:2374] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2. Re-sending same reply.
I20250831 07:58:57.466522 778 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:57.349610 32471 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47886 (request call id 281) took 1878 ms (client timeout 1605 ms). Trace:
I20250831 07:58:57.461717 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:57.467962 32471 rpcz_store.cc:269] 0831 07:58:55.470916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.472473 (+ 1557us) service_pool.cc:225] Handling call
0831 07:58:57.349525 (+1877052us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.468478 31803 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:57.468634 778 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:57.468909 31728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1010) took 2241 ms (client timeout 2980 ms). Trace:
W20250831 07:58:57.469296 31728 rpcz_store.cc:269] 0831 07:58:55.227228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.447117 (+2219889us) service_pool.cc:225] Handling call
0831 07:58:57.468849 (+ 21732us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.469446 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.470244 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.470274 31803 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:57.461726 911 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:57.454252 31773 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:58:57.425292 31800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1355) took 2347 ms (client timeout 2067 ms). Trace:
I20250831 07:58:57.473062 31773 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:57.461673 31801 rpcz_store.cc:276] 0831 07:58:51.686814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:51.926010 (+239196us) service_pool.cc:225] Handling call
0831 07:58:51.938263 (+ 12253us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.440134 (+5501871us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.461029 (+ 20895us) spinlock_profiling.cc:243] Waited 5.42 s on lock 0x7b540003a2d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:57.461228 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12459794816}
I20250831 07:58:57.473227 754 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:58:57.473834 754 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:57.458806 824 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:57.472611 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:57.464107 31733 rpcz_store.cc:276] 0831 07:58:55.214925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.462761 (+2247836us) service_pool.cc:225] Handling call
0831 07:58:57.463643 (+ 882us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.473858 31773 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:57.475236 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.475721 824 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:57.459064 711 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.476073 735 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:57.476409 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.404305 900 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.475592 31773 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:57.477541 711 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:57.407531 31809 rpcz_store.cc:269] 0831 07:58:55.229734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.293423 (+ 63689us) service_pool.cc:225] Handling call
0831 07:58:57.406923 (+2113500us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.462167 31879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1428) took 1998 ms (client timeout 1695 ms). Trace:
I20250831 07:58:57.478118 711 raft_consensus.cc:2802] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:57.478363 900 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.459903 31731 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
W20250831 07:58:57.478168 31879 rpcz_store.cc:269] 0831 07:58:55.463757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.481230 (+ 17473us) service_pool.cc:225] Handling call
0831 07:58:57.462105 (+1980875us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.479080 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1480) took 2472 ms. Trace:
I20250831 07:58:57.479212 31733 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
W20250831 07:58:57.462807 31849 connection.cc:232] RPC call timeout handler was delayed by -1.2716949739999563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1646s for a response
W20250831 07:58:57.356591 32470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47886 (request call id 279) took 1888 ms (client timeout 1967 ms). Trace:
W20250831 07:58:55.372833 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:57.431067 31734 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
W20250831 07:58:57.481951 31734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1013) took 2245 ms (client timeout 2679 ms). Trace:
W20250831 07:58:57.482259 31734 rpcz_store.cc:269] 0831 07:58:55.236119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.429863 (+2193744us) service_pool.cc:225] Handling call
0831 07:58:57.481902 (+ 52039us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.483036 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.484004 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.484189 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:57.484742 31804 raft_consensus.cc:2391] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:58:57.485146 31804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1458) took 2268 ms (client timeout 2249 ms). Trace:
W20250831 07:58:57.485487 31804 rpcz_store.cc:269] 0831 07:58:55.216314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.443540 (+2227226us) service_pool.cc:225] Handling call
0831 07:58:57.485088 (+ 41548us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19712}
I20250831 07:58:57.423688 790 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:57.485984 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:57.471621 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.486546 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.467658 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.474144 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.479573 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:57.486639 32466 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:57.487198 31804 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
W20250831 07:58:57.411518 31802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1362) took 2331 ms (client timeout 2046 ms). Trace:
W20250831 07:58:57.427439 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1015) took 2162 ms (client timeout 2216 ms). Trace:
I20250831 07:58:57.474026 899 raft_consensus.cc:2802] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250831 07:58:57.480110 32470 rpcz_store.cc:269] 0831 07:58:55.468206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:55.468378 (+ 172us) service_pool.cc:225] Handling call
0831 07:58:57.356354 (+1887976us) spinlock_profiling.cc:243] Waited 9.83 ms on lock 0x7b5400484ed4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:57.356519 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22617984}
W20250831 07:58:57.488022 31725 rpcz_store.cc:269] 0831 07:58:55.265513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.422803 (+2157290us) service_pool.cc:225] Handling call
0831 07:58:57.427391 (+ 4588us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.488786 32471 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:57.458442 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.489091 31773 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:57.472865 31800 rpcz_store.cc:269] 0831 07:58:55.078097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.425102 (+2347005us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.425230 (+ 128us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 07:58:57.488160 899 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:57.490025 899 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:57.490195 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.490396 31773 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:57.490784 31800 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
W20250831 07:58:57.479991 31849 connection.cc:232] RPC call timeout handler was delayed by -1.2266596309999613s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21400000000000002s for a response
I20250831 07:58:57.491643 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:57.491987 31849 connection.cc:232] RPC call timeout handler was delayed by -1.2111201870000059s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1932s for a response
I20250831 07:58:57.492208 31800 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.492049 688 consensus_queue.cc:1035] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:58:57.493067 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.493438 907 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:57.487605 31809 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:58:57.493888 900 raft_consensus.cc:2802] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:57.494366 900 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:57.494159 907 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:57.494732 900 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:57.496225 907 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.493695 688 raft_consensus.cc:2802] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:57.496693 907 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:57.496809 688 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:57.497195 688 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:57.497166 907 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:57.499136 907 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.489562 31808 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.485563 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:58:57.450587 31805 rpcz_store.cc:269] 0831 07:58:55.077393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.448881 (+2371488us) service_pool.cc:225] Handling call
0831 07:58:57.450227 (+ 1346us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.476483 735 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:57.487360 31732 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
W20250831 07:58:57.493556 31849 connection.cc:232] RPC call timeout handler was delayed by -1.1707755269999325s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1748s for a response
I20250831 07:58:57.467486 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.486369 790 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.501595 735 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:57.501966 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.502880 790 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:57.479465 31731 rpcz_store.cc:276] 0831 07:58:55.007040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.458956 (+2451916us) service_pool.cc:225] Handling call
0831 07:58:57.479024 (+ 20068us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.433210 31727 rpcz_store.cc:269] 0831 07:58:55.242224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.431376 (+2189152us) service_pool.cc:225] Handling call
0831 07:58:57.432758 (+ 1382us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.503424 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.503926 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.504094 31808 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
W20250831 07:58:57.501430 31849 connection.cc:232] RPC call timeout handler was delayed by -1.0887201869999217s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2068s for a response
W20250831 07:58:57.487766 31802 rpcz_store.cc:269] 0831 07:58:55.079681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.411316 (+2331635us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.411442 (+ 126us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250831 07:58:57.504765 31849 connection.cc:232] RPC call timeout handler was delayed by -1.0527201869999772s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21080000000000002s for a response
I20250831 07:58:57.505004 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.505383 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:57.505558 31808 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:57.505980 31802 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:57.506445 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.506999 31808 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:57.489038 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.404119 771 raft_consensus.cc:2802] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:57.502054 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:57.508462 771 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:57.493438 32434 connection.cc:232] RPC call timeout handler was delayed by -1.3332703080000101s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16620000000000001s for a response
I20250831 07:58:57.508908 771 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:57.509310 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.509863 31732 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:57.508615 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.510552 31725 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:57.510674 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.511282 31732 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:57.511356 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.500030 32463 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:57.415010 704 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.512238 704 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:57.512471 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.512698 704 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.513062 31725 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:57.513630 31731 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:57.513859 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:57.514410 31725 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:57.514626 704 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.515146 704 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:57.515432 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.442159 677 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:57.487388 31801 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.516219 31726 raft_consensus.cc:2391] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:57.516707 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.516767 735 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.517400 31801 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
W20250831 07:58:57.404641 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:57.518085 31803 raft_consensus.cc:2391] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:57.474350 754 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:57.518977 790 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:57.519502 790 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:57.520980 31803 raft_consensus.cc:1273] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:57.521623 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 72) took 4643 ms. Trace:
I20250831 07:58:57.521642 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:58:57.447741 31807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1363) took 2367 ms (client timeout 2666 ms). Trace:
I20250831 07:58:57.479991 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.530400 31733 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:57.531191 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:57.522585 31807 rpcz_store.cc:269] 0831 07:58:55.079931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.446415 (+2366484us) service_pool.cc:225] Handling call
0831 07:58:57.447695 (+ 1280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.532130 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:57.514514 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.533180 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:57.439442 31806 rpcz_store.cc:269] 0831 07:58:55.212711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.438984 (+2226273us) service_pool.cc:212] Skipping call since client already timed out
0831 07:58:57.439074 (+ 90us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 07:58:57.533610 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.533942 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 73) took 4655 ms. Trace:
I20250831 07:58:57.534260 31731 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.403658 31630 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59664 (request call id 213) took 6636 ms. Trace:
I20250831 07:58:57.507745 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:57.471521 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.543768 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.544180 31728 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.544649 31731 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:57.545079 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.545233 31809 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:58:57.545496 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.545675 31728 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:57.546152 31731 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.521658 790 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:57.478632 711 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:57.546913 711 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:57.493611 31800 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:57.529398 32468 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:57.528175 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.504144 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.548965 31727 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:57.549425 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.549770 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.550403 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.550740 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.551370 31727 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:57.517670 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 988) took 4295 ms. Trace:
I20250831 07:58:57.534351 31807 rpcz_store.cc:276] 0831 07:58:52.878809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.532281 (+4653472us) service_pool.cc:225] Handling call
0831 07:58:57.532573 (+ 292us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.532715 (+ 142us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.532736 (+ 21us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.532774 (+ 38us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:58:57.532802 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.532849 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.533892 (+ 1043us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.551975 31726 rpcz_store.cc:276] 0831 07:58:53.222438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.516744 (+4294306us) service_pool.cc:225] Handling call
0831 07:58:57.517024 (+ 280us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.517347 (+ 323us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.517375 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.517422 (+ 47us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:58:57.517460 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.517517 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.517621 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24437,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":121}
I20250831 07:58:57.552923 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:57.553087 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 78) took 4670 ms. Trace:
I20250831 07:58:57.517341 677 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:57.535835 735 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.553740 32468 raft_consensus.cc:2408] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250831 07:58:57.553525 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:57.522167 31803 rpcz_store.cc:276] 0831 07:58:52.878343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.520399 (+4642056us) service_pool.cc:225] Handling call
0831 07:58:57.520693 (+ 294us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.521392 (+ 699us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.521573 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.543871 31808 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 4
I20250831 07:58:57.555231 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1439) took 4353 ms. Trace:
I20250831 07:58:57.556221 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.555596 31803 rpcz_store.cc:276] 0831 07:58:53.201550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.554391 (+4352841us) service_pool.cc:225] Handling call
0831 07:58:57.554701 (+ 310us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.554869 (+ 168us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.554896 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.554959 (+ 63us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:57.554990 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.555064 (+ 74us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.555177 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.556942 31727 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.449833 31729 rpcz_store.cc:276] 0831 07:58:56.355770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.448020 (+1092250us) service_pool.cc:225] Handling call
0831 07:58:57.449465 (+ 1445us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.557735 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:57.558138 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.558678 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.558763 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:57.559125 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.559429 32465 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 3
I20250831 07:58:57.531741 31733 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:57.536895 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:57.560154 32464 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:57.347479 904 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:57.538722 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:57.564004 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:57.564842 904 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:57.578939 31775 leader_election.cc:336] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.731s (SENT)
I20250831 07:58:57.540259 898 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:57.579382 31775 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:57.580106 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:57.580812 32469 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 1.
I20250831 07:58:57.581773 31730 raft_consensus.cc:2391] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:57.543341 31630 rpcz_store.cc:276] 0831 07:58:50.767297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:50.767975 (+ 678us) service_pool.cc:225] Handling call
0831 07:58:57.397860 (+6629885us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:57.397897 (+ 37us) write_op.cc:454] Released schema lock
0831 07:58:57.403602 (+ 5705us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:58:55.477070 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:58:55.486965 (+ 9895us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:58:55.487014 (+ 49us) write_op.cc:435] Acquired schema lock
0831 07:58:55.487035 (+ 21us) tablet.cc:660] Decoding operations
0831 07:58:55.487409 (+ 374us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:58:55.487464 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0831 07:58:55.487494 (+ 30us) tablet.cc:683] Acquiring locks for 1 operations
0831 07:58:55.487686 (+ 192us) tablet.cc:699] Row locks acquired
0831 07:58:55.487709 (+ 23us) write_op.cc:260] PREPARE: finished
0831 07:58:55.487906 (+ 197us) write_op.cc:270] Start()
0831 07:58:55.488032 (+ 126us) write_op.cc:276] Timestamp: P: 1756627135487848 usec, L: 0
0831 07:58:55.488053 (+ 21us) op_driver.cc:348] REPLICATION: starting
0831 07:58:55.488760 (+ 707us) log.cc:844] Serialized 595 byte log entry
0831 07:58:56.185949 (+697189us) op_driver.cc:464] REPLICATION: finished
0831 07:58:56.228901 (+ 42952us) write_op.cc:301] APPLY: starting
0831 07:58:56.229015 (+ 114us) tablet.cc:1365] starting BulkCheckPresence
0831 07:58:56.229136 (+ 121us) tablet.cc:1368] finished BulkCheckPresence
0831 07:58:56.229295 (+ 159us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:58:56.229567 (+ 272us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:58:56.229771 (+ 204us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0831 07:58:56.229833 (+ 62us) write_op.cc:312] APPLY: finished
0831 07:58:56.230109 (+ 276us) log.cc:844] Serialized 23 byte log entry
0831 07:58:56.231103 (+ 994us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:56.231208 (+ 105us) write_op.cc:454] Released schema lock
0831 07:58:56.231258 (+ 50us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":329830,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":42671,"apply.run_cpu_time_us":1969,"apply.run_wall_time_us":2715,"num_ops":1,"prepare.queue_time_us":150245,"prepare.run_cpu_time_us":84728,"prepare.run_wall_time_us":418250,"raft.queue_time_us":73616,"raft.run_cpu_time_us":1803,"raft.run_wall_time_us":43065,"replication_time_us":697702,"thread_start_us":2566810,"threads_started":4,"wal-append.queue_time_us":621106,"wal-append.run_cpu_time_us":12051,"wal-append.run_wall_time_us":1121041}]]}
I20250831 07:58:57.546883 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.584632 31728 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.585467 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:57.565573 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:58:57.597108 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:57.515558 704 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.498987 899 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.621256 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:57.568440 32465 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:57.621794 704 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.569442 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1441) took 4364 ms. Trace:
I20250831 07:58:57.553493 31807 rpcz_store.cc:276] 0831 07:58:52.882505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.552188 (+4669683us) service_pool.cc:225] Handling call
0831 07:58:57.552532 (+ 344us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.552702 (+ 170us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.552738 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.552797 (+ 59us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:58:57.552841 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.552934 (+ 93us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.553033 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.536096 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1436) took 4666 ms. Trace:
I20250831 07:58:57.621794 899 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:57.570907 31698 connection.cc:232] RPC call timeout handler was delayed by -1.0194096750000199s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18520000000000003s for a response
I20250831 07:58:57.623041 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:57.623560 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:57.623705 32465 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:58:57.571146 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:57.624171 31879 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:57.573648 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.568464 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.575628 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:57.624850 914 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:57.575631 31727 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.625279 913 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:57.625388 914 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:57.622335 31803 rpcz_store.cc:276] 0831 07:58:53.204948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.557213 (+4352265us) service_pool.cc:225] Handling call
0831 07:58:57.568964 (+ 11751us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.569127 (+ 163us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.569151 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.569199 (+ 48us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:58:57.569235 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.569289 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.569389 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.626075 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:57.627229 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.627730 914 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:57.624876 31731 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.628496 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:57.629036 31803 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:57.629603 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 63) took 4352 ms. Trace:
I20250831 07:58:57.551402 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.622540 31801 rpcz_store.cc:276] 0831 07:58:52.869246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.519280 (+4650034us) service_pool.cc:225] Handling call
0831 07:58:57.529082 (+ 9802us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.530094 (+ 1012us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.530407 (+ 313us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.530457 (+ 50us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:58:57.530491 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.530548 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.530647 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.630192 31727 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09"
dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:48724
I20250831 07:58:57.630659 31732 raft_consensus.cc:2391] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:57.631047 31801 raft_consensus.cc:1238] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:58:57.631599 31732 raft_consensus.cc:1238] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:57.631678 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1343) took 4203 ms. Trace:
I20250831 07:58:57.529744 778 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:57.631016 31727 raft_consensus.cc:604] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:57.632076 31801 rpcz_store.cc:276] 0831 07:58:53.428722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.630620 (+4201898us) service_pool.cc:225] Handling call
0831 07:58:57.630881 (+ 261us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.631473 (+ 592us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.631625 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.632634 778 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:57.633605 31801 raft_consensus.cc:1238] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 3. Ops: []
I20250831 07:58:57.633900 31727 raft_consensus.cc:1273] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:57.634110 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1345) took 4202 ms. Trace:
I20250831 07:58:57.634459 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 66) took 4356 ms. Trace:
I20250831 07:58:57.634665 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.583452 904 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:57.586719 31847 leader_election.cc:400] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3.
I20250831 07:58:57.621256 31734 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.545681 31725 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:57.587875 31726 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.589192 735 raft_consensus.cc:2747] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250831 07:58:57.592633 31730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1012) took 2361 ms (client timeout 2797 ms). Trace:
I20250831 07:58:57.593521 31775 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:57.635639 31847 leader_election.cc:403] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20250831 07:58:57.638513 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 45090.2us
I20250831 07:58:57.639087 908 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 3
W20250831 07:58:57.638886 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 52303.7us
I20250831 07:58:57.640763 31775 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:57.640930 31627 catalog_manager.cc:5582] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 2, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 2 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:58:57.546296 31809 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:57.566773 32464 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:58:57.485360 902 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.577437 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:57.625766 913 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.642495 31775 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:57.630007 31731 rpcz_store.cc:276] 0831 07:58:53.277173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.628779 (+4351606us) service_pool.cc:225] Handling call
0831 07:58:57.629104 (+ 325us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.629272 (+ 168us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.629303 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.629350 (+ 47us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:58:57.629387 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.629455 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.629554 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.638629 31726 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:57.641515 899 raft_consensus.cc:2802] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:57.646313 31802 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:57.646772 899 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:57.647183 899 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:57.647308 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.635278 31727 rpcz_store.cc:276] 0831 07:58:53.278221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.633385 (+4355164us) service_pool.cc:225] Handling call
0831 07:58:57.633636 (+ 251us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.634252 (+ 616us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.634410 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.644733 900 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.648603 31727 raft_consensus.cc:1238] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:57.649212 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1442) took 4366 ms. Trace:
I20250831 07:58:57.649489 31727 rpcz_store.cc:276] 0831 07:58:53.282507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.648079 (+4365572us) service_pool.cc:225] Handling call
0831 07:58:57.648344 (+ 265us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.649003 (+ 659us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.649164 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.650147 900 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.650545 31727 raft_consensus.cc:1238] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:57.650714 900 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:57.651172 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1443) took 4368 ms. Trace:
I20250831 07:58:57.636343 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.643576 31733 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.652074 31725 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.652949 902 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.653113 31725 raft_consensus.cc:1238] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:57.653422 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1445) took 4367 ms. Trace:
I20250831 07:58:57.653759 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1444) took 4370 ms. Trace:
I20250831 07:58:57.621465 32470 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:57.644690 688 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.654136 31725 rpcz_store.cc:276] 0831 07:58:53.283186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.652640 (+4369454us) service_pool.cc:225] Handling call
0831 07:58:57.652975 (+ 335us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.653565 (+ 590us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.653712 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43264}
I20250831 07:58:57.655491 31725 raft_consensus.cc:1238] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:57.645207 31731 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:57.644922 711 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.657440 778 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:57.658218 31805 raft_consensus.cc:2391] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:57.658962 711 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.659421 31805 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:37044
I20250831 07:58:57.659862 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1441) took 4377 ms. Trace:
I20250831 07:58:57.660498 899 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.661000 31808 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:58:57.661598 31805 raft_consensus.cc:604] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:57.661978 31805 raft_consensus.cc:612] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:58:57.643455 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:57.662365 899 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:58:57.585531 32436 leader_election.cc:336] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.163s (SENT)
I20250831 07:58:57.645784 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 70) took 4366 ms. Trace:
I20250831 07:58:57.663681 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1447) took 4326 ms. Trace:
I20250831 07:58:57.664000 31726 rpcz_store.cc:276] 0831 07:58:53.279419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.637882 (+4358463us) service_pool.cc:225] Handling call
0831 07:58:57.638280 (+ 398us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.645503 (+ 7223us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.645708 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":119296}
I20250831 07:58:57.665534 31726 raft_consensus.cc:1238] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3. Ops: []
W20250831 07:58:57.665762 31773 connection.cc:232] RPC call timeout handler was delayed by -1.107730982000021s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19640000000000002s for a response
I20250831 07:58:57.645656 754 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.666148 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1449) took 4328 ms. Trace:
I20250831 07:58:57.666565 31726 rpcz_store.cc:276] 0831 07:58:53.337761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.665085 (+4327324us) service_pool.cc:225] Handling call
0831 07:58:57.665366 (+ 281us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.665897 (+ 531us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.666098 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.667518 754 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.667752 771 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.646261 735 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:57.672441 31800 raft_consensus.cc:1238] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 4. Ops: []
I20250831 07:58:57.672607 735 raft_consensus.cc:695] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:57.672989 31808 raft_consensus.cc:1273] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:57.673717 31807 raft_consensus.cc:1238] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:57.673782 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 77) took 4793 ms. Trace:
I20250831 07:58:57.673555 735 consensus_queue.cc:237] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.674216 31800 rpcz_store.cc:276] 0831 07:58:52.880010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.547954 (+4667944us) service_pool.cc:225] Handling call
0831 07:58:57.672266 (+124312us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.672816 (+ 550us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.673635 (+ 819us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b540015b150. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:57.673736 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":285219072}
I20250831 07:58:57.674866 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1342) took 4246 ms. Trace:
I20250831 07:58:57.664098 31728 rpcz_store.cc:276] 0831 07:58:53.337148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.662972 (+4325824us) service_pool.cc:225] Handling call
0831 07:58:57.663245 (+ 273us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.663403 (+ 158us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.663427 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.663466 (+ 39us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:57.663496 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.663544 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.663632 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.675209 31807 rpcz_store.cc:276] 0831 07:58:53.428477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.622625 (+4194148us) service_pool.cc:225] Handling call
0831 07:58:57.622913 (+ 288us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.674194 (+ 51281us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.674688 (+ 494us) spinlock_profiling.cc:243] Waited 50.6 ms on lock 0x7b5400029754. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:57.674818 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116479360}
I20250831 07:58:57.676757 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 80) took 3849 ms. Trace:
I20250831 07:58:57.676841 31775 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:57.677217 31807 rpcz_store.cc:276] 0831 07:58:53.827805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.676098 (+3848293us) service_pool.cc:225] Handling call
0831 07:58:57.676366 (+ 268us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.676506 (+ 140us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.676527 (+ 21us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.676562 (+ 35us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:58:57.676589 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.676631 (+ 42us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.676712 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.678051 31775 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:57.655932 688 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.678715 711 raft_consensus.cc:2747] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:58:57.679303 688 raft_consensus.cc:2762] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250831 07:58:57.656030 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1446) took 4369 ms. Trace:
I20250831 07:58:57.678674 31807 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:57.680125 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1451) took 4341 ms. Trace:
I20250831 07:58:57.682176 31808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1443) took 3922 ms. Trace:
I20250831 07:58:57.682658 31808 rpcz_store.cc:276] 0831 07:58:53.751124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.672480 (+3921356us) service_pool.cc:225] Handling call
0831 07:58:57.672728 (+ 248us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.673368 (+ 640us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.673536 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.683216 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:57.675808 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1445) took 3920 ms. Trace:
I20250831 07:58:57.594929 898 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:58:57.655900 31730 rpcz_store.cc:269] 0831 07:58:55.230617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.437407 (+2206790us) service_pool.cc:225] Handling call
0831 07:58:57.591768 (+154361us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54003e2e54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:58:57.591903 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":329288064}
I20250831 07:58:57.668162 754 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:57.684324 754 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.683748 31800 rpcz_store.cc:276] 0831 07:58:53.755959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.675089 (+3919130us) service_pool.cc:225] Handling call
0831 07:58:57.675389 (+ 300us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.675539 (+ 150us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.675564 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.675603 (+ 39us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:57.675630 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.675678 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.675765 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.686754 754 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.686916 31800 raft_consensus.cc:1238] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:57.668373 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1450) took 4330 ms. Trace:
I20250831 07:58:57.683980 31875 raft_consensus.cc:2391] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:57.687300 31726 rpcz_store.cc:276] 0831 07:58:53.338034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.667338 (+4329304us) service_pool.cc:225] Handling call
0831 07:58:57.667700 (+ 362us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.668055 (+ 355us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.668085 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.668130 (+ 45us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:58:57.668159 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.668215 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.668327 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":551,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87}
I20250831 07:58:57.689440 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1452) took 4350 ms. Trace:
I20250831 07:58:57.689788 31726 rpcz_store.cc:276] 0831 07:58:53.338600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.688679 (+4350079us) service_pool.cc:225] Handling call
0831 07:58:57.688955 (+ 276us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.689127 (+ 172us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.689157 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.689197 (+ 40us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:57.689234 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.689287 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.689389 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.653752 31733 rpcz_store.cc:276] 0831 07:58:53.285953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.652709 (+4366756us) service_pool.cc:225] Handling call
0831 07:58:57.652975 (+ 266us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.653114 (+ 139us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.653138 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.653197 (+ 59us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:57.653234 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.653284 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.653372 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20992}
I20250831 07:58:57.691146 31726 raft_consensus.cc:1238] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:57.634493 31801 rpcz_store.cc:276] 0831 07:58:53.431578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.632920 (+4201342us) service_pool.cc:225] Handling call
0831 07:58:57.633457 (+ 537us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.633918 (+ 461us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.634068 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.692544 31801 raft_consensus.cc:1238] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
W20250831 07:58:57.692478 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 68647.2us
I20250831 07:58:57.693174 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1348) took 2616 ms. Trace:
I20250831 07:58:57.693521 31801 rpcz_store.cc:276] 0831 07:58:55.076442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.692091 (+2615649us) service_pool.cc:225] Handling call
0831 07:58:57.692388 (+ 297us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.692985 (+ 597us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.693130 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.660218 31731 rpcz_store.cc:276] 0831 07:58:53.282245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.644712 (+4362467us) service_pool.cc:225] Handling call
0831 07:58:57.645035 (+ 323us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.659595 (+ 14560us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.659781 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.666152 31805 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:37044
I20250831 07:58:57.696228 31805 raft_consensus.cc:604] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:57.696617 31805 raft_consensus.cc:612] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:58:57.697842 31805 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20250831 07:58:57.669481 32435 leader_election.cc:336] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 2.163s (SENT)
I20250831 07:58:57.653944 31727 rpcz_store.cc:276] 0831 07:58:53.282854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.650140 (+4367286us) service_pool.cc:225] Handling call
0831 07:58:57.650387 (+ 247us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.650969 (+ 582us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.651128 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.668789 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 83295.5us
I20250831 07:58:57.691733 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1460) took 3839 ms. Trace:
I20250831 07:58:57.679675 31775 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:57.680337 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 81) took 3836 ms. Trace:
I20250831 07:58:57.648061 31802 raft_consensus.cc:2366] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3.
I20250831 07:58:57.642553 31729 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:57.700042 31807 rpcz_store.cc:276] 0831 07:58:53.843755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.678038 (+3834283us) service_pool.cc:225] Handling call
0831 07:58:57.678410 (+ 372us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.680131 (+ 1721us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.680294 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.700693 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 71) took 4421 ms. Trace:
I20250831 07:58:57.680652 31725 rpcz_store.cc:276] 0831 07:58:53.286533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.655074 (+4368541us) service_pool.cc:225] Handling call
0831 07:58:57.655351 (+ 277us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.655785 (+ 434us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.655985 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.637024 31734 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:57.701610 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 69) took 4422 ms. Trace:
I20250831 07:58:57.701912 31734 rpcz_store.cc:276] 0831 07:58:53.279120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.636541 (+4357421us) service_pool.cc:225] Handling call
0831 07:58:57.636772 (+ 231us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.701377 (+ 64605us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.701559 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.666818 31773 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:57.702420 31775 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:57.691709 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1454) took 4348 ms. Trace:
I20250831 07:58:57.704221 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1354) took 2626 ms. Trace:
I20250831 07:58:57.704232 31726 rpcz_store.cc:276] 0831 07:58:53.343267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.690659 (+4347392us) service_pool.cc:225] Handling call
0831 07:58:57.690952 (+ 293us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.691498 (+ 546us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.691662 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.704491 31802 raft_consensus.cc:1238] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 07:58:57.706228 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1356) took 2627 ms. Trace:
I20250831 07:58:57.642848 31847 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:57.706780 32435 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:57.678974 771 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.707734 677 raft_consensus.cc:2747] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250831 07:58:57.708660 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 62193.8us
I20250831 07:58:57.704591 31805 rpcz_store.cc:276] 0831 07:58:55.077858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.697369 (+2619511us) service_pool.cc:225] Handling call
0831 07:58:57.697645 (+ 276us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:57.704005 (+ 6360us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.704172 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.709462 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:57.706530 31802 rpcz_store.cc:276] 0831 07:58:55.078329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.703981 (+2625652us) service_pool.cc:225] Handling call
0831 07:58:57.704288 (+ 307us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:57.705986 (+ 1698us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.706166 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.710516 32471 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:57.682121 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:57.587348 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1341) took 4159 ms. Trace:
I20250831 07:58:57.712311 31879 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:57.682627 31728 rpcz_store.cc:276] 0831 07:58:53.338327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.675784 (+4337457us) service_pool.cc:225] Handling call
0831 07:58:57.678169 (+ 2385us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.678329 (+ 160us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.678351 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.678392 (+ 41us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:58:57.679133 (+ 741us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.679672 (+ 539us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.679777 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.711328 31804 rpcz_store.cc:276] 0831 07:58:53.428213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.559028 (+4130815us) service_pool.cc:225] Handling call
0831 07:58:57.586679 (+ 27651us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b5800245198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:58:57.586865 (+ 186us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.587025 (+ 160us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.587055 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.587099 (+ 44us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:58:57.587135 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.587197 (+ 62us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.587298 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30374144}
I20250831 07:58:57.714764 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1449) took 2590 ms. Trace:
I20250831 07:58:57.676460 31627 catalog_manager.cc:5582] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 2 to 3, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 3 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:58:57.648079 913 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.715638 754 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:57.715193 31804 rpcz_store.cc:276] 0831 07:58:55.124080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.713845 (+2589765us) service_pool.cc:225] Handling call
0831 07:58:57.714203 (+ 358us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.714385 (+ 182us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.714422 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.714477 (+ 55us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:58:57.714519 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.714588 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.714704 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.716143 754 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:57.716514 754 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:57.716444 31627 catalog_manager.cc:5582] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 2, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 2 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:57.684000 31808 raft_consensus.cc:1238] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:57.718482 31627 catalog_manager.cc:5582] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 2, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 2 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:58:57.687521 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1347) took 2611 ms. Trace:
I20250831 07:58:57.632272 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 65) took 4354 ms. Trace:
I20250831 07:58:57.719854 31808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1346) took 2645 ms. Trace:
I20250831 07:58:57.720213 31808 rpcz_store.cc:276] 0831 07:58:55.074892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.683514 (+2608622us) service_pool.cc:225] Handling call
0831 07:58:57.683795 (+ 281us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.719611 (+ 35816us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.719783 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.720045 31732 rpcz_store.cc:276] 0831 07:58:53.277881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.631222 (+4353341us) service_pool.cc:225] Handling call
0831 07:58:57.631460 (+ 238us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.632071 (+ 611us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.632226 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.721370 31800 rpcz_store.cc:276] 0831 07:58:55.076271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.686457 (+2610186us) service_pool.cc:225] Handling call
0831 07:58:57.686753 (+ 296us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.687281 (+ 528us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.687471 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.722404 899 raft_consensus.cc:2802] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:57.696291 914 consensus_queue.cc:1046] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20250831 07:58:57.706933 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 64151.8us
I20250831 07:58:57.707487 904 raft_consensus.cc:2802] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:57.709672 31805 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:37044
I20250831 07:58:57.645462 890 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.712002 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:57.696566 31731 raft_consensus.cc:1273] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:57.637311 31806 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:57.722949 899 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:57.725601 31847 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:57.734359 32435 leader_election.cc:400] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3.
I20250831 07:58:57.735992 32435 leader_election.cc:403] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250831 07:58:57.736233 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:57.671386 31803 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:58:57.699848 31733 rpcz_store.cc:276] 0831 07:58:53.852325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.691074 (+3838749us) service_pool.cc:225] Handling call
0831 07:58:57.691335 (+ 261us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.691481 (+ 146us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.691504 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:58:57.691546 (+ 42us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:58:57.691574 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.691619 (+ 45us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.691691 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.703634 31773 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:57.738224 31805 raft_consensus.cc:604] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:57.738579 31805 raft_consensus.cc:612] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:58:57.738703 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 74) took 4859 ms. Trace:
I20250831 07:58:57.739107 31806 rpcz_store.cc:276] 0831 07:58:52.879166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.534016 (+4654850us) service_pool.cc:225] Handling call
0831 07:58:57.534319 (+ 303us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.734288 (+199969us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.738325 (+ 4037us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b5400084dd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:57.738645 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":236186368}
I20250831 07:58:57.681957 31809 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:57.728130 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:57.729017 31729 rpcz_store.cc:276] 0831 07:58:53.279706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.637882 (+4358176us) service_pool.cc:225] Handling call
0831 07:58:57.642260 (+ 4378us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.700466 (+ 58206us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.700643 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25344}
I20250831 07:58:57.729915 771 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:57.730362 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:57.730865 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:57.731038 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:57.756426 31802 raft_consensus.cc:1238] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:57.751433 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 76) took 4871 ms. Trace:
I20250831 07:58:57.739559 31773 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
W20250831 07:58:57.752308 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 57909.1us
I20250831 07:58:57.753361 31849 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:57.739315 31627 catalog_manager.cc:5582] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 2, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 2 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:58:57.728809 904 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:57.756570 31883 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:58:57.756884 31809 rpcz_store.cc:276] 0831 07:58:52.879730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.545875 (+4666145us) service_pool.cc:225] Handling call
0831 07:58:57.546187 (+ 312us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.751152 (+204965us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.751376 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:58:57.733047 31775 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:57.759617 31849 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:57.759928 31627 catalog_manager.cc:5582] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 2, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 2 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
W20250831 07:58:57.734774 32436 leader_election.cc:336] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.790s (SENT)
I20250831 07:58:57.761616 31698 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:57.739425 899 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:58:57.763216 31699 connection.cc:232] RPC call timeout handler was delayed by -2.2714760039999646s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1627s for a response
I20250831 07:58:57.732103 909 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.113s
W20250831 07:58:57.895797 31755 debug-util.cc:398] Leaking SignalData structure 0x7b0800191860 after lost signal to thread 30567
I20250831 07:58:57.830762 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:57.817715 32434 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:58:57.898823 31755 debug-util.cc:398] Leaking SignalData structure 0x7b08000c6220 after lost signal to thread 31666
I20250831 07:58:57.840147 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:57.899300 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 81713.4us
I20250831 07:58:57.829638 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:57.900418 31755 debug-util.cc:398] Leaking SignalData structure 0x7b080033b9a0 after lost signal to thread 31758
W20250831 07:58:57.901397 31755 debug-util.cc:398] Leaking SignalData structure 0x7b08003842a0 after lost signal to thread 31759
I20250831 07:58:57.833870 913 raft_consensus.cc:3053] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:58:57.908057 913 raft_consensus.cc:738] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:57.908687 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.795612 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.740615 688 raft_consensus.cc:2802] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:57.755764 771 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:57.831101 677 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20250831 07:58:57.864468 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.913440 771 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:57.913834 31806 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.908915 913 consensus_queue.cc:260] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:57.828544 917 consensus_queue.cc:1046] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.110s
I20250831 07:58:57.915148 913 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:57.843357 31881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7ecb257174554a26b2c35ad04b0eabae"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:58:57.762986 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1462) took 3893 ms. Trace:
W20250831 07:58:57.918004 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:57.828680 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:57.804971 711 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:57.739691 747 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:58:57.806685 919 raft_consensus.cc:2802] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:57.829879 890 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.915661 898 raft_consensus.cc:3053] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:58:57.841877 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.830189 924 raft_consensus.cc:2747] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:58:57.763914 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1361) took 2684 ms. Trace:
I20250831 07:58:57.930142 31802 rpcz_store.cc:276] 0831 07:58:55.079527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.710531 (+2631004us) service_pool.cc:225] Handling call
0831 07:58:57.710807 (+ 276us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.756823 (+ 46016us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.763646 (+ 6823us) spinlock_profiling.cc:243] Waited 45.4 ms on lock 0x7b54001d9554. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:58:57.763857 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":104504192}
I20250831 07:58:57.830467 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:58:57.912112 31699 connection.cc:232] RPC call timeout handler was delayed by -2.2651760039999544s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1634s for a response
W20250831 07:58:57.931715 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:57.932137 31699 connection.cc:232] RPC call timeout handler was delayed by -2.2354760039999064s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16670000000000001s for a response
I20250831 07:58:57.932420 31805 raft_consensus.cc:2391] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:58:57.932684 31699 connection.cc:232] RPC call timeout handler was delayed by -2.1282953080000198s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
I20250831 07:58:57.902617 909 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250831 07:58:57.933526 31699 connection.cc:232] RPC call timeout handler was delayed by -1.9065835050000715s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18520000000000003s for a response
W20250831 07:58:57.933998 31699 connection.cc:232] RPC call timeout handler was delayed by -1.7432960950000052s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20070000000000002s for a response
W20250831 07:58:57.934221 31699 connection.cc:232] RPC call timeout handler was delayed by -1.7338896499999237s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20670000000000002s for a response
W20250831 07:58:57.934440 31699 connection.cc:232] RPC call timeout handler was delayed by -1.5621769329999324s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
I20250831 07:58:57.846611 908 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3.
I20250831 07:58:57.916461 31881 raft_consensus.cc:604] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Received request to transfer leadership to TS 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:57.806941 824 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:57.935854 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1353) took 2858 ms. Trace:
I20250831 07:58:57.935962 790 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:57.936455 824 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:57.936476 31881 raft_consensus.cc:612] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:58:57.937143 824 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:57.938277 790 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:57.938889 790 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:57.936553 31801 rpcz_store.cc:276] 0831 07:58:55.077629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.694199 (+2616570us) service_pool.cc:225] Handling call
0831 07:58:57.694489 (+ 290us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:57.694829 (+ 340us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.694864 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:58:57.695320 (+ 456us) log.cc:844] Serialized 26 byte log entry
0831 07:58:57.737294 (+ 41974us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:58:57.935212 (+197918us) log.cc:844] Serialized 14 byte log entry
0831 07:58:57.935527 (+ 315us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.935599 (+ 72us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:58:57.935637 (+ 38us) raft_consensus.cc:1683] finished
0831 07:58:57.935648 (+ 11us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:57.935771 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1024,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"thread_start_us":218682,"threads_started":2,"wal-append.queue_time_us":44442}
I20250831 07:58:57.824693 32436 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:57.940618 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 205889us
I20250831 07:58:57.909626 31627 catalog_manager.cc:5582] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 2, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 2 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:57.918682 31882 raft_consensus.cc:2374] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4. Re-sending same reply.
I20250831 07:58:57.829988 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:57.942304 31875 raft_consensus.cc:2374] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3. Re-sending same reply.
I20250831 07:58:57.739719 907 raft_consensus.cc:2747] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3.
W20250831 07:58:57.919471 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 91689.4us
I20250831 07:58:57.909591 31809 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.946278 824 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.953519 824 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:57.830212 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.911518 904 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:57.917680 31731 rpcz_store.cc:276] 0831 07:58:53.865056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.695908 (+3830852us) service_pool.cc:225] Handling call
0831 07:58:57.696237 (+ 329us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:58:57.734103 (+ 37866us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:57.758272 (+ 24169us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:58:57.765017 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:57.827194 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 67642.2us
W20250831 07:58:57.958020 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:57.929612 898 raft_consensus.cc:738] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:57.958320 31849 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:57.958770 898 consensus_queue.cc:260] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.959407 31849 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:57.959786 908 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:57.959981 898 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:58:57.960515 908 raft_consensus.cc:2802] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:57.960866 908 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:57.961093 31849 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:57.961917 31849 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:57.962839 31849 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:57.911253 688 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:57.926180 919 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:57.926447 711 raft_consensus.cc:695] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:58:57.806798 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 73841.7us
I20250831 07:58:57.939668 31805 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
mode: GRACEFUL
new_leader_uuid: "a0d845d01c8c4894895f66e3beae298c"
from {username='slave'} at 127.0.0.1:37044
I20250831 07:58:57.906661 754 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.967396 31775 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:57.966629 740 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:57.968255 31775 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:58:57.934640 31699 connection.cc:232] RPC call timeout handler was delayed by -1.5040552669998988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1675s for a response
W20250831 07:58:57.968942 31699 connection.cc:232] RPC call timeout handler was delayed by -1.3948982859999433s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19030000000000002s for a response
W20250831 07:58:57.969405 31699 connection.cc:232] RPC call timeout handler was delayed by -1.3273760039999161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2676s for a response
W20250831 07:58:57.969821 31699 connection.cc:232] RPC call timeout handler was delayed by -1.1567741270000624s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20750000000000002s for a response
I20250831 07:58:57.970132 31775 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:57.935552 909 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:57.971349 31699 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:58:57.908700 31755 debug-util.cc:398] Leaking SignalData structure 0x7b08003b6720 after lost signal to thread 31833
I20250831 07:58:57.975354 890 raft_consensus.cc:2802] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:57.975754 890 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:57.976156 890 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:57.968815 911 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:57.980422 911 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:57.980186 917 consensus_queue.cc:1035] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:57.980628 778 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:57.980818 911 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:57.981072 778 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:57.966845 919 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:57.981487 778 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:58:57.982223 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:57.982638 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:57.983561 31755 debug-util.cc:398] Leaking SignalData structure 0x7b0800132ec0 after lost signal to thread 31907
W20250831 07:58:57.983942 31755 debug-util.cc:398] Leaking SignalData structure 0x7b080032e7c0 after lost signal to thread 31908
I20250831 07:58:57.983896 890 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:57.984459 711 consensus_queue.cc:237] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:57.985320 31755 debug-util.cc:398] Leaking SignalData structure 0x7b0800399fe0 after lost signal to thread 32495
W20250831 07:58:57.985611 31755 debug-util.cc:398] Leaking SignalData structure 0x7b0800399fa0 after lost signal to thread 32496
I20250831 07:58:57.988292 31699 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:58:57.924664 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 163120us
I20250831 07:58:57.924623 747 raft_consensus.cc:695] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:57.961177 908 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:57.760838 31773 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:57.987906 735 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250831 07:58:57.994047 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 233269us
I20250831 07:58:57.943264 31627 catalog_manager.cc:5582] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 2 to 3, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 3 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:58:57.927275 31800 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:57.967407 31805 raft_consensus.cc:604] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Received request to transfer leadership to TS a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:57.996398 31805 raft_consensus.cc:612] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:58:57.972333 31775 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:57.996191 31627 catalog_manager.cc:5582] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 2, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 2 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250831 07:58:57.997570 31775 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:58:57.972364 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:57.999840 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.000615 31878 raft_consensus.cc:2408] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:58:58.002229 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:58.006304 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 269624us
I20250831 07:58:58.006759 31627 catalog_manager.cc:5582] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 2 to 3, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 3 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:58:57.973269 909 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.008855 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:57.989560 31698 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:58.008791 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.009718 31883 raft_consensus.cc:2391] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:57.990619 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.010007 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.010341 31875 raft_consensus.cc:2391] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:58.011220 31698 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:58.012334 31698 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:58.005770 735 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:58.012586 747 consensus_queue.cc:237] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.013578 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.014263 31876 raft_consensus.cc:2391] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:58.014678 31698 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.014979 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:57.992034 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.016023 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.012375 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:57.963802 688 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.008479 31627 catalog_manager.cc:5582] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 2, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 2 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:58:57.992115 917 raft_consensus.cc:2762] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250831 07:58:58.013126 735 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:57.968941 824 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 07:58:58.017311 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.003028 31877 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
W20250831 07:58:58.018587 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.019891 711 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:57.986111 890 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:57.988776 31879 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:58:58.021085 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.047322 31808 raft_consensus.cc:2391] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:58.009166 900 raft_consensus.cc:2747] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:58.016788 31882 raft_consensus.cc:2408] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250831 07:58:58.021720 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 121379us
I20250831 07:58:58.021648 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.021641 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20250831 07:58:58.048592 31883 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.049923 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:58:58.050922 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:58.022574 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.038522 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.042295 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.015637 924 raft_consensus.cc:2802] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:58.045039 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.089095 31877 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:58.009050 902 raft_consensus.cc:2802] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250831 07:58:58.021273 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 58491.2us
I20250831 07:58:58.020699 914 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250831 07:58:58.092902 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 84831.3us
I20250831 07:58:58.020876 704 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:58.089860 31875 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:58.100365 31699 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:58.107002 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.049062 711 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:58.113648 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.117969 31877 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.021962 929 raft_consensus.cc:2762] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250831 07:58:58.052304 917 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.132022 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:58:58.133319 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 135817us
I20250831 07:58:58.089119 924 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:58.093034 902 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:58.138211 31879 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.150488 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.151281 31875 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.154278 31848 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:58.155345 31879 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:58.125623 711 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:58.137681 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 89374.9us
I20250831 07:58:58.093237 930 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:58.157296 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.161324 32434 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:58.163705 31883 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:58.166757 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.167426 31883 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.165498 31875 raft_consensus.cc:2408] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:58:58.180956 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:58.181985 31877 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:58.185225 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.138013 917 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:58.189889 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.193888 31877 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:58.196734 31848 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:58.200202 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 44402.7us
I20250831 07:58:58.112941 904 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.156814 902 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.185995 31875 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.187435 31883 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:58.148118 914 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250831 07:58:58.188871 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 88650.8us
I20250831 07:58:58.156116 924 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.186807 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:58:58.215991 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.225841 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:58.226627 31699 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.227806 31699 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:58.226876 32470 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:58.232084 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.234467 31883 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.231019 31773 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:58.235783 31877 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:58:58.236444 940 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.232453 31879 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:58.237399 824 raft_consensus.cc:2747] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:58.182229 938 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:58.211673 917 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.244846 31880 raft_consensus.cc:2391] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 2: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:58.161172 771 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.221256 914 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:58.233695 754 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.240419 704 raft_consensus.cc:2802] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.242156 934 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.253221 31883 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:58.255195 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.254490 704 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:58.256088 704 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:58:58.256177 931 log.cc:927] Time spent T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
I20250831 07:58:58.257325 740 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:58:58.257573 920 log.cc:927] Time spent T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
W20250831 07:58:58.259330 933 log.cc:927] Time spent T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20250831 07:58:58.261641 790 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.262553 771 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:58.262485 911 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.265635 735 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.266744 899 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.267525 735 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.267627 790 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:58.268646 919 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.270838 778 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.271303 711 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.272634 778 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.272545 704 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.273239 711 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.273778 31883 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.274641 704 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.275746 902 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.277619 902 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:58.278110 919 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.278107 31883 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:58.279102 908 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.252614 914 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.280565 908 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:58.272634 904 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:58.283560 688 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.284318 914 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.285320 899 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:58.285367 911 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:58.285964 688 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:58.286327 914 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:58.287717 949 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250831 07:58:58.323732 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 308068us
W20250831 07:58:58.329493 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 101638us
I20250831 07:58:58.330858 31699 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:58.340827 31809 raft_consensus.cc:2391] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:58.344004 31803 raft_consensus.cc:2391] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:58.253845 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.344617 31699 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:58.247951 31875 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:58.375509 914 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.376211 914 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:58.376678 914 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.252733 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.345937 31879 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:58.388751 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:58.390125 31878 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:58.390986 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:58.386273 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:58.287899 32435 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:58.387202 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:58.387493 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:58.379047 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:58:58.386330 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.375392 913 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.400190 943 raft_consensus.cc:2747] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:58.401247 32471 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.403554 32468 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.404383 929 consensus_queue.cc:1035] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:58.398900 913 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:58.408246 913 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:58:58.413564 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.254657 934 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:58.414520 32463 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.416450 934 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:58:58.419974 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 247221us
W20250831 07:58:58.420867 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 132930us
I20250831 07:58:58.251765 940 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:58.421918 940 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.425268 907 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:58.426084 907 raft_consensus.cc:695] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 3 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:58:58.398452 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 59931.2us
I20250831 07:58:58.427994 31775 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.416221 31849 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.434803 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1360) took 3336 ms. Trace:
I20250831 07:58:58.436160 938 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:58:58.435314 31807 rpcz_store.cc:276] 0831 07:58:55.079356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.710608 (+2631252us) service_pool.cc:225] Handling call
0831 07:58:57.710883 (+ 275us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:58:57.725203 (+ 14320us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:58:57.725235 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:58:57.725779 (+ 544us) log.cc:844] Serialized 26 byte log entry
0831 07:58:57.993454 (+267675us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:58:58.021812 (+ 28358us) log.cc:844] Serialized 14 byte log entry
0831 07:58:58.022006 (+ 194us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:58:58.022105 (+ 99us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:58:58.273995 (+251890us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b1800452980. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b792 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:58:58.415131 (+141136us) raft_consensus.cc:1683] finished
0831 07:58:58.415364 (+ 233us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:58:58.415670 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17293,"prepare.queue_time_us":24126,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":5656,"raft.queue_time_us":93141,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":6819,"spinlock_wait_cycles":9589376,"thread_start_us":237143,"threads_started":1,"wal-append.queue_time_us":306604}
I20250831 07:58:58.427136 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.441943 31731 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:58.444039 31849 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.432828 907 consensus_queue.cc:237] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.445254 904 raft_consensus.cc:2762] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250831 07:58:58.446038 31849 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:58.446269 904 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.446807 904 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:58.447276 904 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.448951 31849 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.452309 908 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.452569 911 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.452941 908 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:58.453287 911 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:58.454981 911 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.453505 908 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.461290 32471 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:58.464443 31773 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:58.464515 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.465811 688 raft_consensus.cc:2802] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:58.466450 688 raft_consensus.cc:695] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:58.467782 688 consensus_queue.cc:237] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.470233 909 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.473188 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.474366 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.475147 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.475497 31729 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:58:58.476069 31732 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:58.476962 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.477288 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.479027 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.479262 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.480252 31726 raft_consensus.cc:2366] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3.
I20250831 07:58:58.479885 31728 raft_consensus.cc:2466] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:58:58.480960 913 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:58.481967 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 48869.5us
W20250831 07:58:58.482148 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.452575 31731 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:58:58.485508 911 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.488095 911 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:58.488306 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:58:58.488974 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.489295 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:58.490042 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.495400 31732 raft_consensus.cc:2374] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4. Re-sending same reply.
W20250831 07:58:58.490198 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 66515.2us
I20250831 07:58:58.490602 919 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.496136 919 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:58.496866 919 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.501080 31775 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:58.504714 32471 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 07:58:58.490980 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.492146 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.492969 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.493716 913 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.494196 31807 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20250831 07:58:58.492180 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.516799 31728 raft_consensus.cc:2374] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3. Re-sending same reply.
I20250831 07:58:58.517472 890 raft_consensus.cc:2802] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:58.517910 890 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:58.518399 890 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.519017 778 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:58:58.520942 31731 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:58:58.525142 31729 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
W20250831 07:58:58.523952 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 62921us
I20250831 07:58:58.506759 909 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.538559 949 consensus_queue.cc:1046] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250831 07:58:58.540594 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.540437 938 raft_consensus.cc:3053] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:58.541158 938 raft_consensus.cc:738] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:58.542680 938 consensus_queue.cc:260] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.543627 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.544402 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.544682 31729 raft_consensus.cc:2408] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 07:58:58.545650 938 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.546736 31733 raft_consensus.cc:1273] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:58.550848 31733 raft_consensus.cc:1238] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
W20250831 07:58:58.552852 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:58.556391 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.557402 924 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:58.558313 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 349049us
I20250831 07:58:58.564754 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.565088 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.565326 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.565794 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.566129 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.566696 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.567605 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.567545 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.568830 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.528587 914 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.531157 908 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.588617 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.589337 914 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:58.589874 908 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:58.590031 31731 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:58.591248 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:58.591634 924 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.592653 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.593326 31732 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.593567 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.593503 31848 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:58.594347 31731 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.594406 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.595155 31726 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.572892 917 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.577797 934 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.579705 31698 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:58.584861 32468 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:58.582885 31733 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:58.584926 32463 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
W20250831 07:58:58.585367 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.585979 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.587157 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.577714 919 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.604265 31729 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:58.607784 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.609030 954 raft_consensus.cc:2802] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:58.609647 954 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:58.610059 954 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.609122 31734 raft_consensus.cc:1238] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:58.613121 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.613361 31728 raft_consensus.cc:1273] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:58:58.614198 31773 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:58.607405 31847 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:58.617249 941 raft_consensus.cc:2802] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:58.617827 941 raft_consensus.cc:695] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:58.618472 934 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:58.619303 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.620191 31727 raft_consensus.cc:2391] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:58.620785 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:58.621562 31698 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:58.622686 941 consensus_queue.cc:237] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.623852 929 raft_consensus.cc:2802] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:58.624655 929 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:58.624874 31732 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:58.625130 929 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.625530 31698 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.626046 940 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.623929 688 raft_consensus.cc:2802] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:58.627399 688 raft_consensus.cc:695] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:58.628880 904 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:58.632035 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.632728 31800 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:58.634840 917 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.635558 688 consensus_queue.cc:237] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.639469 31698 leader_election.cc:400] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Vote denied by peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585) with higher term. Message: Invalid argument: T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3.
I20250831 07:58:58.640686 31698 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.641438 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.642578 917 raft_consensus.cc:2802] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.643093 917 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:58.643646 917 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.644233 914 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:58:58.644325 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.644739 914 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:58.630728 924 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:58.645606 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.646763 31849 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.647114 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.647301 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:58.647742 911 raft_consensus.cc:2802] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.646782 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.648407 911 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:58.648960 911 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.645187 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.651005 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.651558 31725 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 4
I20250831 07:58:58.648978 31730 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:58.652460 31698 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.653465 31848 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:58:58.654136 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 47315.8us
I20250831 07:58:58.641458 890 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.653553 940 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:58.647786 31734 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:58.659130 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.660209 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.660967 31698 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:58.645102 914 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:58.665894 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.669116 942 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:58.669543 31727 raft_consensus.cc:2391] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:58.671701 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.675323 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:58.680079 31733 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:58.665289 917 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.682428 31726 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:58.685325 31725 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:58:58.676435 908 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:58.685953 924 raft_consensus.cc:2762] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250831 07:58:58.681970 904 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:58.695174 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.695467 890 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.696038 31881 raft_consensus.cc:2391] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:58.697031 31849 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.697958 919 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:58.699569 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:58.691169 934 consensus_queue.cc:1035] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20250831 07:58:58.700695 917 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.701548 934 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:58.702129 934 raft_consensus.cc:695] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:58.700330 32471 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:58:58.689801 908 raft_consensus.cc:695] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:58.708050 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.714066 31878 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:58.726889 31877 raft_consensus.cc:1238] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
W20250831 07:58:58.724347 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.721328 32470 raft_consensus.cc:1238] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:58.720424 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:58.729036 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.719802 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.728507 31878 raft_consensus.cc:1238] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:58.742322 31882 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:58:58.720593 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:58.703310 934 consensus_queue.cc:237] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.725135 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:58:58.728643 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:58.719967 32463 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:58.753211 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.754084 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:58.759332 32464 raft_consensus.cc:2408] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250831 07:58:58.760273 31879 raft_consensus.cc:2391] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:58.753262 32465 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
W20250831 07:58:58.762858 957 log.cc:927] Time spent T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20250831 07:58:58.749951 954 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.771229 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.774740 932 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:58:58.775223 31773 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:58:58.775476 917 consensus_queue.cc:1046] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:58:58.775470 914 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.776304 919 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:58.777055 917 raft_consensus.cc:3053] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:58.777456 917 raft_consensus.cc:738] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:58.777712 919 raft_consensus.cc:695] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:58.778110 914 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.778419 917 consensus_queue.cc:260] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.790088 31698 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.796466 31731 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:58.797623 31849 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.798321 954 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:58.798681 31849 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:58.799032 954 raft_consensus.cc:2802] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.798804 932 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:58:58.800012 941 raft_consensus.cc:2802] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.800447 941 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:58.799026 919 consensus_queue.cc:237] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.800856 941 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.800710 911 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.802701 911 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:58.794592 914 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:58:58.799510 954 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:58:58.804003 914 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:58.804216 954 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.804463 914 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:58.805814 31848 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:58.806722 932 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:58:58.807554 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.809403 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:58.810122 31808 raft_consensus.cc:2374] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4. Re-sending same reply.
I20250831 07:58:58.812836 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.813534 31808 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:58.817929 790 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:58.818523 790 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:58.818015 31878 raft_consensus.cc:1238] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:58.826138 31698 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:58:58.834590 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.821117 31881 raft_consensus.cc:1238] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:58.824626 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.843895 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.844100 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:58.847002 31876 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:58.847551 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.824164 790 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.789799 908 consensus_queue.cc:237] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.853978 31882 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:58.847235 924 raft_consensus.cc:2802] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250831 07:58:58.852759 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 52936us
I20250831 07:58:58.858402 31808 raft_consensus.cc:2391] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:58:58.855796 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.856204 917 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.857487 924 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:58.865423 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.865901 31880 raft_consensus.cc:1238] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:58.869379 924 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.873246 31805 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:58.874449 941 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.865103 954 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.877362 954 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:58.878810 924 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.883544 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.884475 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.885718 924 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.885843 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.886626 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.887328 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.887768 31733 raft_consensus.cc:2391] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:58.888197 31727 raft_consensus.cc:2391] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:58.889003 941 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:58.869472 909 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20250831 07:58:58.890599 929 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.887338 31725 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.900470 953 consensus_queue.cc:1046] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:58.900964 32434 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:58.868371 942 consensus_queue.cc:1046] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20250831 07:58:58.905690 938 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:58:58.906111 938 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:58.906490 938 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:58.918401 31725 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:58.920845 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.921892 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.916059 913 raft_consensus.cc:3053] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:58.922945 31725 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:58.924342 32436 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:58.925012 909 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:58.925801 32436 leader_election.cc:400] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Vote denied by peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027) with higher term. Message: Invalid argument: T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3.
I20250831 07:58:58.926733 32436 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:58.927376 909 raft_consensus.cc:2802] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:58:58.927778 909 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:58.928159 909 raft_consensus.cc:3058] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.929337 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.930039 31807 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:58.930985 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:58.931249 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.931361 913 raft_consensus.cc:738] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:58.932134 31807 raft_consensus.cc:2366] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3.
I20250831 07:58:58.932188 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:58.932382 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.932914 31805 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.932467 913 consensus_queue.cc:260] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.933331 31802 raft_consensus.cc:2366] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3.
I20250831 07:58:58.933805 913 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.934101 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.935457 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.936199 32434 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:58.929986 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.937244 31803 raft_consensus.cc:2391] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:58.937740 32434 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:58.937907 932 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:58.936192 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.931594 31809 raft_consensus.cc:2374] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5. Re-sending same reply.
I20250831 07:58:58.944725 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.946069 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.946730 31806 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:58.947875 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.948225 31808 raft_consensus.cc:2391] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:58:58.948910 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:58.950449 939 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:58:58.950752 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.951074 939 raft_consensus.cc:695] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:58.951493 31801 raft_consensus.cc:2366] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3.
I20250831 07:58:58.951278 704 consensus_queue.cc:1046] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:58.952448 704 raft_consensus.cc:3053] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:58.952179 939 consensus_queue.cc:237] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.952816 704 raft_consensus.cc:738] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:58.952817 31727 raft_consensus.cc:1238] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20250831 07:58:58.954053 31731 raft_consensus.cc:1238] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:58.953691 704 consensus_queue.cc:260] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.954917 704 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.963099 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.951084 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.968434 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.968609 32434 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:58.969177 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.969956 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.972688 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.973603 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.972520 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.975000 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.948633 31803 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:58.976478 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.978348 31730 raft_consensus.cc:2366] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3.
I20250831 07:58:58.981220 31805 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:58.982012 31732 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.982057 31728 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:58.983144 31734 raft_consensus.cc:2391] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:58.982249 31727 raft_consensus.cc:2374] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3. Re-sending same reply.
I20250831 07:58:58.984579 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:58.982800 31800 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:58.987532 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:58.989037 31807 raft_consensus.cc:2391] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:58.991318 951 consensus_queue.cc:1046] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
I20250831 07:58:58.991643 938 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.992931 898 raft_consensus.cc:3053] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:58.993309 898 raft_consensus.cc:738] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:58.994098 938 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:58.994138 898 consensus_queue.cc:260] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.994760 790 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.995553 898 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:58.983795 953 consensus_queue.cc:1046] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20250831 07:58:59.004807 953 raft_consensus.cc:3053] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:59.005133 953 raft_consensus.cc:738] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:59.005960 953 consensus_queue.cc:260] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:58.987437 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:59.007192 953 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:59.008383 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:59.009064 31880 raft_consensus.cc:2391] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:59.010597 31809 raft_consensus.cc:2391] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:58.987167 914 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:58.996419 31773 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:58.965956 934 consensus_queue.cc:1035] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250831 07:58:58.987540 907 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:59.000180 929 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:59.011559 31728 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:59.017987 31808 raft_consensus.cc:2391] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:58:59.023907 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 49248us
I20250831 07:58:59.016561 31732 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:59.019559 31729 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:59.022148 31882 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:58:59.025672 31808 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 07:58:59.017289 31849 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.021199 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.027925 31806 raft_consensus.cc:2391] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:59.028589 31803 raft_consensus.cc:2391] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:59.029349 31698 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:59.030321 913 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:58:59.030845 913 raft_consensus.cc:695] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:59.031975 913 consensus_queue.cc:237] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.032753 32436 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.033545 932 raft_consensus.cc:2747] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:59.034444 32436 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:59.035308 932 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:58:59.035560 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:59.035933 907 raft_consensus.cc:695] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 3 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:59.035697 932 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:59.036345 31732 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.036631 932 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:59.037094 907 consensus_queue.cc:237] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.035640 32436 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:59.041460 32434 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:59.047663 31729 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:59.049053 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:58:59.050735 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 54450us
I20250831 07:58:59.070947 31732 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:59.076494 31729 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 07:58:59.081329 31800 raft_consensus.cc:1238] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
W20250831 07:58:59.084858 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:59.086813 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.087579 31806 raft_consensus.cc:2391] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:58:59.088220 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 46842.2us
I20250831 07:58:59.080749 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.090093 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.051481 919 raft_consensus.cc:2747] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:59.098018 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.102496 31808 raft_consensus.cc:1238] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
W20250831 07:58:59.104506 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:59.071619 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:59.076232 949 raft_consensus.cc:2747] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 07:58:59.110348 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 93179.6us
I20250831 07:58:59.111140 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.072125 934 raft_consensus.cc:2747] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:58:59.112722 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.116612 31806 raft_consensus.cc:2391] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:59.118697 974 raft_consensus.cc:2762] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250831 07:58:59.118907 31807 raft_consensus.cc:1238] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:59.119560 32436 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.119796 31773 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:59.119889 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.120560 31805 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:58:59.120903 31874 raft_consensus.cc:2366] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3.
I20250831 07:58:59.121297 974 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:58:59.117147 31849 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:59.119932 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:58:59.122165 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:59.122495 31876 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:59.123549 31698 leader_election.cc:400] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Vote denied by peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585) with higher term. Message: Invalid argument: T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3.
I20250831 07:58:59.123920 31698 leader_election.cc:403] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250831 07:58:59.125324 32434 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.085908 938 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
W20250831 07:58:59.171186 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 68072.8us
W20250831 07:58:59.172084 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 55018us
I20250831 07:58:59.174609 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:59.127807 704 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.118002 790 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:59.128851 914 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:59.127982 940 raft_consensus.cc:2747] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:59.205016 704 raft_consensus.cc:2747] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585) with higher term. Message: Invalid argument: T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3.
W20250831 07:58:59.205442 982 log.cc:927] Time spent T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250831 07:58:59.208844 917 raft_consensus.cc:3053] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:59.209285 917 raft_consensus.cc:738] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:59.210140 917 consensus_queue.cc:260] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.212189 909 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.212517 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.213207 31804 raft_consensus.cc:2391] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:59.214169 909 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:59.215371 32463 raft_consensus.cc:1273] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:59.216069 31773 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:59.198765 890 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:59.216430 31849 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:59.203063 932 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.219689 31876 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:58:59.211599 917 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.216071 32465 raft_consensus.cc:1238] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:59.230713 31698 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:58:59.231412 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 106196us
I20250831 07:58:59.235219 31879 raft_consensus.cc:1238] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:59.236863 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.237361 31806 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
W20250831 07:58:59.238231 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 65351.5us
I20250831 07:58:59.231751 939 raft_consensus.cc:2747] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:59.230784 704 consensus_queue.cc:1046] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20250831 07:58:59.180266 954 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:59.243755 31849 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:59.248147 32464 raft_consensus.cc:1238] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:59.246604 934 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20250831 07:58:59.246636 890 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.250090 790 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:59.250594 790 raft_consensus.cc:695] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:59.250763 913 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:58:59.251266 890 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:59.251443 790 consensus_queue.cc:237] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.252787 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:58:59.253929 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.254575 31806 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:59.255522 929 consensus_queue.cc:1046] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:58:59.256338 929 consensus_queue.cc:1230] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250831 07:58:59.257510 983 raft_consensus.cc:3053] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:59.257941 983 raft_consensus.cc:738] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:59.257416 31630 catalog_manager.cc:5582] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 3, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 3 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:59.258875 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:59.258843 983 consensus_queue.cc:260] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:58:59.260066 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 44078.7us
I20250831 07:58:59.261461 31773 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.257871 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:59.263437 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:59.264927 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:59.265314 31732 raft_consensus.cc:2391] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:59.265834 31729 raft_consensus.cc:2391] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:59.251593 953 raft_consensus.cc:3053] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:59.255300 932 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:59.268157 31730 raft_consensus.cc:1238] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:59.259088 31630 catalog_manager.cc:5582] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 2, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 2 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:59.244886 940 raft_consensus.cc:2747] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:59.299357 31727 raft_consensus.cc:1238] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:59.299150 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.299801 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:59.303885 31879 raft_consensus.cc:1238] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:59.303948 31732 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:59.301929 31734 raft_consensus.cc:1238] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 07:58:59.309315 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.313661 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.314929 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.269012 953 raft_consensus.cc:738] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:59.268532 890 raft_consensus.cc:2747] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:58:59.264601 913 raft_consensus.cc:695] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:59.318698 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.322727 31807 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.324319 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.324090 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.310282 983 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
W20250831 07:58:59.326730 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 65354.1us
I20250831 07:58:59.326718 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.328194 31805 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:59.310518 904 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:59.329139 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.328388 31808 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:58:59.335047 31809 raft_consensus.cc:1238] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 07:58:59.307684 932 consensus_queue.cc:1046] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20250831 07:58:59.341662 31773 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.340467 934 consensus_queue.cc:1046] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20250831 07:58:59.342777 953 consensus_queue.cc:260] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.345621 31630 catalog_manager.cc:5582] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 2, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 2 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:59.353391 31630 catalog_manager.cc:5582] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 3, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 3 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:58:59.341636 932 raft_consensus.cc:3053] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:59.359856 31874 raft_consensus.cc:1238] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:59.369525 932 raft_consensus.cc:738] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:59.370656 932 consensus_queue.cc:260] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.361406 32463 raft_consensus.cc:1238] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:58:59.375907 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.376152 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.376765 31801 raft_consensus.cc:2366] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4.
I20250831 07:58:59.377084 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.377539 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.377868 31806 raft_consensus.cc:2391] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:59.378269 31801 raft_consensus.cc:2391] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:59.378969 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.379603 31801 raft_consensus.cc:2391] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:59.363082 992 raft_consensus.cc:3053] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:59.339175 985 consensus_queue.cc:1046] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20250831 07:58:59.367995 913 consensus_queue.cc:237] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.338402 904 raft_consensus.cc:695] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:59.380752 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.390457 31806 raft_consensus.cc:2391] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:59.390355 31732 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
W20250831 07:58:59.381336 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:59.381734 31773 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.383144 31807 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:58:59.392306 32436 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:58:59.386404 31800 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:58:59.380393 992 raft_consensus.cc:738] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:59.395879 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.398716 31630 catalog_manager.cc:5582] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:58:59.401371 31849 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.402118 941 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:58:59.402562 31849 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:59.403281 941 raft_consensus.cc:2802] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:59.403379 31849 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:59.403851 941 raft_consensus.cc:695] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:59.404156 911 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:59.404814 911 raft_consensus.cc:695] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:59.404686 941 consensus_queue.cc:237] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.406095 911 consensus_queue.cc:237] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.408560 941 consensus_queue.cc:1046] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:59.410086 941 consensus_queue.cc:1046] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:59.411136 941 consensus_queue.cc:1046] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:59.419273 31627 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59674 (request call id 214) took 2096 ms. Trace:
I20250831 07:58:59.424280 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.424618 31807 raft_consensus.cc:1238] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:59.430330 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:59.430652 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:58:59.430949 31733 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:59.431257 31732 raft_consensus.cc:2366] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4.
I20250831 07:58:59.432142 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:58:59.432175 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 169437us
I20250831 07:58:59.432773 31734 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.434573 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.437079 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:58:59.435482 31800 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:59.461673 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:58:59.462054 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 80438us
I20250831 07:58:59.462592 31802 raft_consensus.cc:2391] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:58:59.461864 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 69754.8us
W20250831 07:58:59.464658 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:59.467773 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 62461.7us
I20250831 07:58:59.470745 31775 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.472491 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.473433 31727 raft_consensus.cc:1238] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:58:59.469190 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:58:59.469784 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.477316 31806 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:59.477226 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:59.478212 31879 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.470969 31773 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.492103 31732 raft_consensus.cc:1273] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:59.492533 31804 raft_consensus.cc:2391] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:58:59.425240 688 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:58:59.497665 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.498716 31802 raft_consensus.cc:2366] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3.
I20250831 07:58:59.389645 917 consensus_queue.cc:1046] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.087s
I20250831 07:58:59.501829 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:59.501952 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:58:59.504091 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 92884.4us
I20250831 07:58:59.469450 938 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:58:59.504644 31801 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.506995 31876 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250831 07:58:59.507669 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:59.507332 988 log.cc:927] Time spent T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20250831 07:58:59.515246 31880 raft_consensus.cc:1238] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:59.503922 908 raft_consensus.cc:3053] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:59.497140 688 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:59.501578 932 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.492972 890 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:58:59.501763 954 raft_consensus.cc:695] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:59.521466 31800 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
W20250831 07:58:59.514607 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 43957.1us
I20250831 07:58:59.522584 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:59.530469 31775 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.530848 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:59.524544 32464 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:58:59.536835 31882 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:58:59.520076 31874 raft_consensus.cc:1238] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:59.544871 31727 raft_consensus.cc:1273] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:58:59.545200 31733 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:58:59.519830 938 raft_consensus.cc:695] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:59.521306 908 raft_consensus.cc:738] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:59.546062 31730 raft_consensus.cc:1238] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:59.522531 688 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:59.520043 940 raft_consensus.cc:2747] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:58:59.432608 31627 rpcz_store.cc:276] 0831 07:58:57.323207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.323938 (+ 731us) service_pool.cc:225] Handling call
0831 07:58:59.418999 (+2095061us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:58:58.528335 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:58:58.585133 (+ 56798us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:58:58.585180 (+ 47us) write_op.cc:435] Acquired schema lock
0831 07:58:58.585212 (+ 32us) tablet.cc:660] Decoding operations
0831 07:58:58.586460 (+ 1248us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:58:58.586569 (+ 109us) write_op.cc:641] Partition lock acquired for write op
0831 07:58:58.586608 (+ 39us) tablet.cc:683] Acquiring locks for 11 operations
0831 07:58:58.600919 (+ 14311us) tablet.cc:699] Row locks acquired
0831 07:58:58.604695 (+ 3776us) write_op.cc:260] PREPARE: finished
0831 07:58:58.609649 (+ 4954us) write_op.cc:270] Start()
0831 07:58:58.609821 (+ 172us) write_op.cc:276] Timestamp: P: 1756627138609566 usec, L: 0
0831 07:58:58.609859 (+ 38us) op_driver.cc:348] REPLICATION: starting
0831 07:58:58.610981 (+ 1122us) log.cc:844] Serialized 4365 byte log entry
0831 07:58:58.900653 (+289672us) op_driver.cc:464] REPLICATION: finished
0831 07:58:59.050491 (+149838us) write_op.cc:301] APPLY: starting
0831 07:58:59.050598 (+ 107us) tablet.cc:1365] starting BulkCheckPresence
0831 07:58:59.071497 (+ 20899us) tablet.cc:1368] finished BulkCheckPresence
0831 07:58:59.071642 (+ 145us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:58:59.106698 (+ 35056us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:58:59.106976 (+ 278us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0831 07:58:59.107035 (+ 59us) write_op.cc:312] APPLY: finished
0831 07:58:59.107348 (+ 313us) log.cc:844] Serialized 83 byte log entry
0831 07:58:59.107523 (+ 175us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:59.107649 (+ 126us) write_op.cc:454] Released schema lock
0831 07:58:59.107711 (+ 62us) write_op.cc:341] FINISH: Updating metrics
0831 07:58:59.108192 (+ 481us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:58:59.108209 (+ 17us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1067,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":149519,"apply.run_cpu_time_us":3780,"apply.run_wall_time_us":57710,"num_ops":11,"prepare.queue_time_us":395392,"prepare.run_cpu_time_us":23743,"prepare.run_wall_time_us":159508,"raft.queue_time_us":112456,"raft.run_cpu_time_us":16094,"raft.run_wall_time_us":73377,"replication_time_us":290574,"thread_start_us":500774,"threads_started":4,"wal-append.queue_time_us":81417}]]}
I20250831 07:58:59.553246 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:59.557291 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:59.558382 1001 raft_consensus.cc:3053] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:59.558795 1001 raft_consensus.cc:738] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:59.514029 904 consensus_queue.cc:237] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.559441 1002 raft_consensus.cc:3053] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Stepping down as leader of term 2
W20250831 07:58:59.558835 999 log.cc:927] Time spent T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20250831 07:58:59.564590 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:59.566275 31877 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:59.568655 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:59.568580 908 consensus_queue.cc:260] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:58:59.569669 987 log.cc:927] Time spent T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.203s user 0.003s sys 0.004s
I20250831 07:58:59.569662 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:59.582183 31801 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:58:59.513813 992 consensus_queue.cc:260] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.551512 890 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.571390 1002 raft_consensus.cc:738] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:59.592386 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:58:59.595731 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:59.584067 940 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 07:58:59.597622 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 126770us
I20250831 07:58:59.607156 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:59.599500 31874 raft_consensus.cc:2391] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:58:59.632570 938 consensus_queue.cc:237] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.636018 31878 raft_consensus.cc:1238] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:59.636777 998 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:58:59.637862 31699 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:59.637518 1002 consensus_queue.cc:260] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.638592 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:59.639998 890 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:58:59.645871 31879 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:58:59.647778 31773 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:58:59.648478 31734 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:58:59.596297 992 raft_consensus.cc:3058] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:59.662339 917 consensus_queue.cc:1046] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:58:59.649281 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 80393.3us
I20250831 07:58:59.601047 940 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:58:59.654882 31800 raft_consensus.cc:1238] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:59.655206 980 raft_consensus.cc:2747] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:58:59.655436 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.655776 1007 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:59.656077 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.669147 31804 raft_consensus.cc:2391] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:58:59.600749 1001 consensus_queue.cc:260] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.595338 954 consensus_queue.cc:237] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.661422 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:58:59.661674 31699 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:59.673599 704 raft_consensus.cc:3053] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:59.674515 1005 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:59.674772 704 raft_consensus.cc:738] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:59.674988 890 raft_consensus.cc:2802] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:59.675518 890 raft_consensus.cc:695] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:59.675704 704 consensus_queue.cc:260] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.675140 1005 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.677929 1002 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.678102 32434 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.678365 949 consensus_queue.cc:1046] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:58:59.679998 907 raft_consensus.cc:3053] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:59.680941 31728 raft_consensus.cc:1273] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:58:59.672577 940 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:58:59.708251 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 101173us
W20250831 07:58:59.712550 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:58:59.716602 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 55006.7us
I20250831 07:58:59.719930 704 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:59.720062 980 consensus_queue.cc:1046] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:58:59.698422 1012 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:58:59.701323 907 raft_consensus.cc:738] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:59.720511 31883 raft_consensus.cc:1273] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:58:59.724992 31802 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:58:59.724020 31803 raft_consensus.cc:1273] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:58:59.737071 32466 raft_consensus.cc:1238] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:58:59.676467 890 consensus_queue.cc:237] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.729700 688 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.738570 898 consensus_queue.cc:1046] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:58:59.739382 688 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:59.730921 31801 raft_consensus.cc:1238] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:59.685451 929 raft_consensus.cc:3053] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:58:59.685452 1007 consensus_queue.cc:1035] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20250831 07:58:59.740418 929 raft_consensus.cc:738] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:59.741271 929 consensus_queue.cc:260] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.742389 31804 raft_consensus.cc:1238] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:58:59.743083 929 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:58:59.733428 1005 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:59.743717 907 consensus_queue.cc:260] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.745048 31699 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:59.745513 907 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.746248 31699 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:59.747403 31699 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:59.748579 917 raft_consensus.cc:3053] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:58:59.748966 917 raft_consensus.cc:738] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:58:59.749766 917 consensus_queue.cc:260] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.732369 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:58:59.751005 917 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:59.752413 31775 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.718493 983 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:59.753347 1005 raft_consensus.cc:2747] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:59.755209 31775 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:58:59.754222 996 log.cc:927] Time spent T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.191s user 0.001s sys 0.000s
I20250831 07:58:59.756637 953 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:59.757603 1005 raft_consensus.cc:2802] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:59.757844 31775 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:59.758074 1005 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:58:59.758447 1005 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.758708 978 raft_consensus.cc:2762] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250831 07:58:59.759361 978 raft_consensus.cc:2802] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:59.759675 978 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:58:59.760010 978 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.763160 1001 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.684478 1008 consensus_queue.cc:1035] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20250831 07:58:59.768910 908 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:58:59.775003 909 raft_consensus.cc:3053] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:59.775434 909 raft_consensus.cc:738] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:59.774775 974 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20250831 07:58:59.776418 909 consensus_queue.cc:260] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.778349 949 consensus_queue.cc:1046] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:58:59.778733 1007 raft_consensus.cc:2762] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250831 07:58:59.779309 939 raft_consensus.cc:3053] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:59.779649 939 raft_consensus.cc:738] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:58:59.784925 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 148923us
I20250831 07:58:59.785445 31775 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:59.785888 31773 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:59.787809 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:59.806604 31775 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.819797 31880 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:58:59.817804 31848 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
W20250831 07:58:59.824476 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 77156.5us
I20250831 07:58:59.831528 31699 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:59.833072 31773 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.818571 980 raft_consensus.cc:2747] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:58:59.822475 909 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:58:59.843971 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:58:59.824717 1007 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:58:59.794481 914 consensus_queue.cc:1035] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
W20250831 07:58:59.848270 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 139483us
I20250831 07:58:59.851459 31775 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:58:59.830925 1008 raft_consensus.cc:2802] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:59.853341 31699 leader_election.cc:400] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4.
I20250831 07:58:59.859380 31699 leader_election.cc:403] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250831 07:58:59.842702 904 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:58:59.859229 913 raft_consensus.cc:2747] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:58:59.804126 939 consensus_queue.cc:260] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.851727 998 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:59.866171 31848 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.864246 951 raft_consensus.cc:2747] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4.
I20250831 07:58:59.859100 932 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20250831 07:58:59.866003 904 raft_consensus.cc:695] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:58:59.889717 31699 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:59.895479 31699 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:58:59.896413 31699 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:59.897812 31699 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:58:59.881831 939 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.866894 1022 consensus_queue.cc:1046] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20250831 07:58:59.897338 951 raft_consensus.cc:2747] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:58:59.915248 1023 raft_consensus.cc:3053] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:58:59.915642 1023 raft_consensus.cc:738] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:58:59.917405 1023 consensus_queue.cc:260] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.919363 1023 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.898013 914 raft_consensus.cc:2747] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:58:59.903025 942 raft_consensus.cc:2747] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:58:59.877188 998 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.905892 31876 raft_consensus.cc:2466] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:58:59.881971 1002 raft_consensus.cc:2802] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:58:59.931245 1002 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:58:59.907955 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 55792.6us
I20250831 07:58:59.932909 998 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:58:59.933457 31848 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.908350 913 raft_consensus.cc:2747] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:59.935041 31848 leader_election.cc:400] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3.
I20250831 07:58:59.935979 31848 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.937091 31848 leader_election.cc:400] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3.
I20250831 07:58:59.937580 978 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.909608 940 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.938405 908 consensus_queue.cc:1035] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:58:59.910544 904 consensus_queue.cc:237] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:59.940356 908 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:59.931669 1002 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.947425 31882 consensus_queue.cc:237] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:58:59.964882 31848 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.947786 983 raft_consensus.cc:2747] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:59.954067 978 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:59.954720 940 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:59.975517 31728 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:58:59.971465 914 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:59.971027 909 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:59.976508 908 raft_consensus.cc:2747] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:58:59.987798 31726 raft_consensus.cc:1238] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: []
I20250831 07:58:59.988210 1011 raft_consensus.cc:2747] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:58:59.988416 890 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:58:59.988932 1011 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:59.989641 978 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:59.989563 1011 raft_consensus.cc:695] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:59.987891 31734 raft_consensus.cc:1238] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20250831 07:58:59.990381 978 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.988976 890 raft_consensus.cc:695] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:59.992359 890 consensus_queue.cc:237] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.993790 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:58:59.979548 919 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:58:59.974133 929 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:58:59.989475 1007 raft_consensus.cc:695] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:58:59.995457 978 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:58:59.995045 929 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:58:59.995642 1007 consensus_queue.cc:237] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:58:59.996646 993 raft_consensus.cc:2762] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250831 07:58:59.969384 954 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
W20250831 07:59:00.005831 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 177557us
W20250831 07:59:00.006240 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 57517.8us
I20250831 07:59:00.006862 31773 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:58:59.977159 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:58:59.964412 998 consensus_queue.cc:1046] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20250831 07:59:00.010342 31734 raft_consensus.cc:1238] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:59:00.012707 31848 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.010867 31874 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:58:59.997509 914 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:00.014750 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 163357us
I20250831 07:59:00.015247 688 raft_consensus.cc:2747] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.016045 1002 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.017103 914 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:00.017175 31775 leader_election.cc:400] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Vote denied by peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027) with higher term. Message: Invalid argument: T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3.
I20250831 07:59:00.018082 1002 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:00.018781 1005 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:00.024214 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:00.044344 31734 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:59:00.025549 31876 raft_consensus.cc:1238] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:00.047638 31730 raft_consensus.cc:1273] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:59:00.048892 917 consensus_queue.cc:1046] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:59:00.050714 917 raft_consensus.cc:3053] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:00.026338 31775 leader_election.cc:403] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250831 07:59:00.053678 929 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:00.054317 31775 leader_election.cc:400] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Vote denied by peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027) with higher term. Message: Invalid argument: T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4.
I20250831 07:59:00.003986 909 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:00.039057 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:00.052237 31874 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
W20250831 07:59:00.042685 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:00.054214 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:00.056701 31880 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:00.033654 31773 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.058682 31698 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:00.059886 31698 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:00.069146 31728 raft_consensus.cc:1273] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:00.051123 917 raft_consensus.cc:738] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:00.078004 31805 raft_consensus.cc:1238] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:00.078145 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:00.079157 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:00.102368 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 89732.7us
I20250831 07:59:00.103070 31848 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.072027 951 raft_consensus.cc:2802] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:58:59.991200 1011 consensus_queue.cc:237] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.077236 688 raft_consensus.cc:2747] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027) with higher term. Message: Invalid argument: T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3.
I20250831 07:59:00.058050 938 consensus_queue.cc:1046] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20250831 07:59:00.081175 998 raft_consensus.cc:2747] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:00.077427 914 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:00.054754 939 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20250831 07:59:00.076215 940 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20250831 07:59:00.112031 31848 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.109485 31849 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:00.072240 1008 raft_consensus.cc:695] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:00.099699 942 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:00.104234 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 48642.7us
I20250831 07:59:00.118368 31880 raft_consensus.cc:1273] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250831 07:59:00.107457 1032 consensus_queue.cc:1046] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20250831 07:59:00.128571 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:00.117512 31775 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:00.152712 31734 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:00.113230 914 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:00.156381 31699 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:00.163128 31732 raft_consensus.cc:1238] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 07:59:00.166657 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:00.155578 32436 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:00.112897 951 raft_consensus.cc:695] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:00.121681 909 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:00.171172 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 137640us
I20250831 07:59:00.114534 938 consensus_queue.cc:1230] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer a0d845d01c8c4894895f66e3beae298c log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250831 07:59:00.172784 31773 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.113706 1007 raft_consensus.cc:2747] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:59:00.177266 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:00.130443 908 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:00.121333 997 raft_consensus.cc:3053] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:59:00.113762 890 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20250831 07:59:00.132043 954 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:00.184475 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:00.188990 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:00.172011 998 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:00.180625 939 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:00.185374 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:00.135474 1005 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:00.169128 919 raft_consensus.cc:2802] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:59:00.182221 953 raft_consensus.cc:2747] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:00.132339 993 consensus_queue.cc:1035] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20250831 07:59:00.188081 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:00.183468 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:00.163954 914 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:00.185287 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 73355.4us
I20250831 07:59:00.193750 31801 raft_consensus.cc:1238] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:00.193768 31804 raft_consensus.cc:1273] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:00.132048 904 consensus_queue.cc:1035] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.052s
I20250831 07:59:00.195279 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:00.201388 31806 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:00.185087 908 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:00.116550 917 consensus_queue.cc:260] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.126858 983 consensus_queue.cc:1035] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20250831 07:59:00.187340 31699 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:00.197539 31808 raft_consensus.cc:2391] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:00.129091 32466 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250831 07:59:00.197429 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:00.209661 31807 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:00.199314 31809 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:00.195456 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:00.188146 997 raft_consensus.cc:738] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:00.121651 995 consensus_queue.cc:1046] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.095s
I20250831 07:59:00.191196 954 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:00.118196 942 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.206261 938 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:00.121495 1018 consensus_queue.cc:1046] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.127s
I20250831 07:59:00.207103 31848 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.207263 1007 raft_consensus.cc:2802] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:59:00.197331 939 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:00.210820 31800 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:00.198772 998 raft_consensus.cc:695] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:00.205004 908 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:00.208549 917 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:00.213112 953 raft_consensus.cc:2747] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.211931 919 raft_consensus.cc:695] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:00.232707 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 49750.6us
I20250831 07:59:00.237636 31699 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:00.238868 31699 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:00.214558 954 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:00.215179 31731 raft_consensus.cc:1238] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20250831 07:59:00.227610 1033 log.cc:927] Time spent T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20250831 07:59:00.229497 939 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:00.250916 31800 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:00.206149 951 consensus_queue.cc:237] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.128109 1008 consensus_queue.cc:237] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.250881 978 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.212551 31880 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:00.259564 998 consensus_queue.cc:237] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.259578 31880 raft_consensus.cc:604] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:00.262053 919 consensus_queue.cc:237] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.263021 890 consensus_queue.cc:1046] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:59:00.264318 1005 raft_consensus.cc:3053] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:00.264678 1005 raft_consensus.cc:738] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:00.246026 992 raft_consensus.cc:2747] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.265754 31699 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:00.265542 1005 consensus_queue.cc:260] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.266606 1005 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:00.211524 940 consensus_queue.cc:1046] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20250831 07:59:00.238631 1007 raft_consensus.cc:695] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:00.223984 938 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:00.273023 31699 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:00.257326 704 raft_consensus.cc:2747] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.283769 1010 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20250831 07:59:00.272537 983 raft_consensus.cc:2747] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.292212 31773 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.293421 1039 raft_consensus.cc:3053] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:00.287396 1007 consensus_queue.cc:237] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.287890 890 raft_consensus.cc:3053] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:00.285280 954 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.295079 1002 raft_consensus.cc:3053] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
W20250831 07:59:00.296371 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 96354.3us
I20250831 07:59:00.300573 31699 leader_election.cc:400] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 election: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3.
I20250831 07:59:00.303359 31699 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:00.301092 704 raft_consensus.cc:2747] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.270221 997 consensus_queue.cc:260] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.305665 31726 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e"
dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:48724
I20250831 07:59:00.314388 31773 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.319348 951 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:00.320683 31848 leader_election.cc:400] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3.
I20250831 07:59:00.306565 997 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:59:00.323808 31699 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:00.301965 1002 raft_consensus.cc:738] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:00.300220 890 raft_consensus.cc:738] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:00.300068 1039 raft_consensus.cc:738] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:00.314064 919 raft_consensus.cc:2802] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:59:00.329212 919 raft_consensus.cc:695] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:00.329764 31848 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.330142 919 consensus_queue.cc:237] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.330920 31848 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.331688 992 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:00.320662 704 raft_consensus.cc:2802] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:00.332089 992 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:00.332567 992 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:00.332312 890 consensus_queue.cc:260] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.333277 942 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:00.333532 890 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:00.335162 938 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:00.328580 31803 raft_consensus.cc:1238] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 1. Current term is 2. Ops: []
I20250831 07:59:00.332526 704 raft_consensus.cc:695] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:00.324986 940 raft_consensus.cc:2802] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:59:00.335933 898 raft_consensus.cc:2747] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.336422 940 raft_consensus.cc:695] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:00.336333 1002 consensus_queue.cc:260] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.336655 704 consensus_queue.cc:237] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.337638 1002 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:59:00.337437 940 consensus_queue.cc:237] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.339123 1036 raft_consensus.cc:3053] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:00.339311 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:00.340421 31848 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.328004 31773 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.339552 1036 raft_consensus.cc:738] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:00.340111 31878 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:59:00.322180 31726 raft_consensus.cc:604] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:00.347976 31726 raft_consensus.cc:612] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:59:00.344942 31698 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:00.345126 31699 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:00.346434 1044 raft_consensus.cc:2747] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:00.347445 954 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:00.350442 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:00.351434 31800 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:00.320008 951 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.352065 31699 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:00.351594 31848 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.353271 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:00.356397 913 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:00.357204 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:00.360394 31773 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.342715 938 consensus_queue.cc:1046] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20250831 07:59:00.361805 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:00.365629 31729 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:00.345629 998 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:00.348671 898 consensus_queue.cc:1035] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20250831 07:59:00.366744 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:00.367329 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:00.368180 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:00.371361 31800 raft_consensus.cc:1238] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:00.354091 942 raft_consensus.cc:2802] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:00.342485 1036 consensus_queue.cc:260] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:00.353204 1018 raft_consensus.cc:2802] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:59:00.383316 31848 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.359182 978 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:00.356808 913 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:00.348502 983 consensus_queue.cc:1046] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20250831 07:59:00.375223 1010 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:59:00.375536 998 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:00.394644 31725 raft_consensus.cc:1273] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:00.397553 31729 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:00.385372 942 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:00.423712 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:00.387262 978 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:00.392019 913 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:00.436705 1036 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:00.437429 917 consensus_queue.cc:1046] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20250831 07:59:00.387120 1008 raft_consensus.cc:2747] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250831 07:59:00.500144 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:00.382699 1039 consensus_queue.cc:260] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:00.500450 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:00.500689 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:00.555022 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:00.555094 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:00.533011 1047 log.cc:927] Time spent T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.121s user 0.000s sys 0.005s
I20250831 07:59:00.553580 31878 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:00.504055 998 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:00.503875 1018 raft_consensus.cc:695] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:00.534283 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 151059us
I20250831 07:59:00.557256 31878 raft_consensus.cc:604] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:00.503378 942 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:00.533818 978 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:00.535356 904 raft_consensus.cc:2747] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:00.563505 31848 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.569499 951 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:00.569867 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:00.570672 1018 consensus_queue.cc:237] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.571485 1010 raft_consensus.cc:2747] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:59:00.572721 1046 log.cc:927] Time spent T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.182s user 0.000s sys 0.000s
I20250831 07:59:00.573436 31808 raft_consensus.cc:1238] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:00.581454 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:00.582687 914 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:00.584929 1042 log.cc:927] Time spent T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.158s user 0.000s sys 0.005s
I20250831 07:59:00.585827 31734 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0"
dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
mode: GRACEFUL
new_leader_uuid: "8247a38b550649aa9cd6e7668dcaec10"
from {username='slave'} at 127.0.0.1:48724
I20250831 07:59:00.586387 31734 raft_consensus.cc:604] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Received request to transfer leadership to TS 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:00.586699 31734 raft_consensus.cc:612] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:59:00.589656 31880 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:00.590164 31880 raft_consensus.cc:604] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:00.590489 31880 raft_consensus.cc:612] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:59:00.591629 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20250831 07:59:00.591975 1045 log.cc:927] Time spent T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
I20250831 07:59:00.594403 998 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.596693 914 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:00.600672 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 44009.6us
I20250831 07:59:00.601929 31773 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.602428 908 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.603020 998 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:00.605649 974 raft_consensus.cc:3053] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:00.606071 974 raft_consensus.cc:738] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:00.565955 1049 log.cc:927] Time spent T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.207s user 0.000s sys 0.005s
I20250831 07:59:00.606909 974 consensus_queue.cc:260] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:00.608014 1032 raft_consensus.cc:2747] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.608409 974 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:00.609225 31773 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.611389 1032 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.613019 31773 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:00.616251 992 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.617954 1031 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:59:00.619071 992 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:00.626467 31878 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:59:00.631290 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:59:00.575516 32464 raft_consensus.cc:3053] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:00.646955 31698 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:00.655529 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:00.672408 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 109017us
I20250831 07:59:00.676057 31848 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.677963 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:00.636596 1039 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:00.684748 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:00.640820 1010 raft_consensus.cc:2747] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:00.642766 704 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:00.695637 32469 raft_consensus.cc:1273] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 07:59:00.642318 32464 raft_consensus.cc:738] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:00.698966 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 319835us
I20250831 07:59:00.700129 32436 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.703539 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:00.704128 31809 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:00.673652 953 raft_consensus.cc:3053] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:00.707479 31848 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.686358 983 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:00.690623 938 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:00.715242 31849 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:00.715162 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:00.704465 704 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.709263 919 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20250831 07:59:00.711082 953 raft_consensus.cc:738] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:00.687376 1032 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:00.703226 954 raft_consensus.cc:2747] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:00.727779 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:59:00.718976 904 consensus_queue.cc:1035] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20250831 07:59:00.709278 985 raft_consensus.cc:2747] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:00.738402 31809 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
W20250831 07:59:00.736380 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 89553.3us
I20250831 07:59:00.719702 32466 raft_consensus.cc:1273] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:59:00.731034 914 raft_consensus.cc:2802] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:00.728502 898 raft_consensus.cc:3053] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
W20250831 07:59:00.761379 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:00.745918 998 raft_consensus.cc:2747] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:00.748881 32463 raft_consensus.cc:1273] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:59:00.749774 1013 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election won for term 3
W20250831 07:59:00.775640 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 68886.7us
I20250831 07:59:00.732090 983 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:00.775923 31773 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.557s (SENT) [suppressed 102 similar messages]
I20250831 07:59:00.759367 914 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:00.759725 898 raft_consensus.cc:738] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:00.785641 31848 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.766568 908 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:00.738452 938 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:00.789340 31732 raft_consensus.cc:1273] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 07:59:00.794021 898 consensus_queue.cc:260] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.796022 983 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:00.801041 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:00.739985 1032 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:00.811604 31809 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:00.769863 939 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:00.790213 914 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:00.789314 1066 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:00.798875 938 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:00.817750 1010 consensus_queue.cc:1035] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:00.818099 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:00.744449 32464 consensus_queue.cc:260] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:00.824294 32465 raft_consensus.cc:1238] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:00.825608 939 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:00.828552 704 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:00.831694 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:00.834755 31878 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
W20250831 07:59:00.839558 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 54031.3us
W20250831 07:59:00.838519 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 60828.6us
I20250831 07:59:00.780440 953 consensus_queue.cc:260] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.840657 31698 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:00.841812 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:00.797784 913 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:00.843019 31809 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:00.814918 1013 raft_consensus.cc:695] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:00.846580 1013 consensus_queue.cc:237] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:00.847419 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:00.857327 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 81417.3us
I20250831 07:59:00.857926 1002 raft_consensus.cc:2747] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.844250 32464 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:00.870745 31630 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43950 (request call id 120) took 2166 ms. Trace:
I20250831 07:59:00.837821 1023 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:00.882305 31848 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.883533 31848 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:00.888228 1018 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:00.888785 1018 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:00.892225 919 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:00.892977 919 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:00.834751 942 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.871209 31630 rpcz_store.cc:276] 0831 07:58:58.704685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:58.708061 (+ 3376us) service_pool.cc:225] Handling call
0831 07:59:00.870657 (+2162596us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:58:59.520098 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:58:59.576536 (+ 56438us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:58:59.581716 (+ 5180us) write_op.cc:435] Acquired schema lock
0831 07:58:59.581748 (+ 32us) tablet.cc:660] Decoding operations
0831 07:58:59.640514 (+ 58766us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:58:59.640572 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0831 07:58:59.640601 (+ 29us) tablet.cc:683] Acquiring locks for 5 operations
0831 07:58:59.641132 (+ 531us) tablet.cc:699] Row locks acquired
0831 07:58:59.641150 (+ 18us) write_op.cc:260] PREPARE: finished
0831 07:58:59.641284 (+ 134us) write_op.cc:270] Start()
0831 07:58:59.641392 (+ 108us) write_op.cc:276] Timestamp: P: 1756627139641248 usec, L: 0
0831 07:58:59.641410 (+ 18us) op_driver.cc:348] REPLICATION: starting
0831 07:58:59.655941 (+ 14531us) log.cc:844] Serialized 2054 byte log entry
0831 07:58:59.871851 (+215910us) op_driver.cc:464] REPLICATION: finished
0831 07:59:00.093287 (+221436us) write_op.cc:301] APPLY: starting
0831 07:59:00.098045 (+ 4758us) tablet.cc:1365] starting BulkCheckPresence
0831 07:59:00.118950 (+ 20905us) tablet.cc:1368] finished BulkCheckPresence
0831 07:59:00.177863 (+ 58913us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:59:00.245853 (+ 67990us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:59:00.280807 (+ 34954us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0831 07:59:00.288072 (+ 7265us) write_op.cc:312] APPLY: finished
0831 07:59:00.293296 (+ 5224us) log.cc:844] Serialized 47 byte log entry
0831 07:59:00.293568 (+ 272us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:00.293680 (+ 112us) write_op.cc:454] Released schema lock
0831 07:59:00.293723 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0831 07:59:00.294100 (+ 377us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:00.294127 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":83092,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":193091,"apply.run_cpu_time_us":3590,"apply.run_wall_time_us":216898,"num_ops":5,"prepare.queue_time_us":103133,"prepare.run_cpu_time_us":4894,"prepare.run_wall_time_us":150723,"replication_time_us":219614,"thread_start_us":158728,"threads_started":2}]]}
W20250831 07:59:00.910373 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 69795.5us
I20250831 07:59:00.916599 31880 raft_consensus.cc:1238] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:00.894537 1010 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.927109 31809 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:00.910041 1002 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.931218 32435 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:00.836902 1066 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:00.933566 31848 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.909861 1018 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:00.916432 974 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:00.896257 890 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
W20250831 07:59:00.937693 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:00.947096 32471 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:00.939381 31848 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.935267 31628 catalog_manager.cc:5582] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 2 to 3, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 3 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:59:00.940019 938 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:00.940707 1032 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:00.894569 1023 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:00.948499 31848 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:00.949949 995 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:00.950806 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:00.951414 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:00.951514 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:00.952363 31727 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250831 07:59:00.955171 31725 raft_consensus.cc:1238] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:00.938025 1010 raft_consensus.cc:2762] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20250831 07:59:01.008809 998 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:00.995684 31731 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:00.990690 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:00.996961 1074 raft_consensus.cc:2802] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:01.013936 1074 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:01.014355 1074 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:01.015340 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:00.990504 938 raft_consensus.cc:695] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:01.016645 31876 raft_consensus.cc:1238] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:00.990185 954 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:00.938575 917 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:01.016970 1023 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:00.999716 1071 log.cc:927] Time spent T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20250831 07:59:01.000131 31628 catalog_manager.cc:5582] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 2, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:59:01.004091 31800 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20250831 07:59:01.022961 31800 pending_rounds.cc:85] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 5
I20250831 07:59:01.023262 31800 pending_rounds.cc:107] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20250831 07:59:01.023530 31800 raft_consensus.cc:2887] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:01.004997 992 raft_consensus.cc:991] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10: : Instructing follower 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:59:01.005580 1077 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:00.941792 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:00.957744 1018 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20250831 07:59:00.995337 913 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:01.030155 954 raft_consensus.cc:1079] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Signalling peer 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:59:01.015496 909 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:01.037654 909 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.037998 1077 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.018438 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:00.987154 995 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.017360 917 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.018335 998 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.020709 31878 raft_consensus.cc:1273] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:00.955902 974 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.039795 909 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:01.040751 1077 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:01.042115 995 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:01.045923 974 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:01.042164 917 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:01.022042 1010 consensus_queue.cc:1035] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20250831 07:59:01.042361 942 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:01.041915 919 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:01.022310 941 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20250831 07:59:01.048550 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:01.016986 938 consensus_queue.cc:237] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.049176 32467 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:01.049536 951 consensus_queue.cc:1046] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:59:01.029841 939 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:01.050437 939 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.042484 998 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:01.051713 978 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.052438 939 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:01.052983 942 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:01.052832 1074 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.044445 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:01.053728 978 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:01.053504 942 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.054383 31730 raft_consensus.cc:1238] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:01.054721 1074 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:01.021723 31727 raft_consensus.cc:1215] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Deduplicated request from leader. Original: 1.4->[3.5-3.5] Dedup: 3.5->[]
I20250831 07:59:01.042809 1066 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:01.055298 31727 raft_consensus.cc:1238] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250831 07:59:01.056200 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:01.048847 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:01.056939 31809 raft_consensus.cc:2391] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:01.057027 31805 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:59:01.031376 31628 catalog_manager.cc:5582] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 2, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:59:01.057868 953 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:01.058423 32435 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:01.052863 31698 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:01.027248 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:01.060271 31726 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:00.944348 1002 consensus_queue.cc:1046] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
I20250831 07:59:01.046542 32463 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:00.941509 1072 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:01.062673 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:01.067807 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.064131 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:00.956072 1013 raft_consensus.cc:2747] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:01.064730 31699 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:01.068027 31725 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:01.064021 31848 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:01.070629 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:01.073871 31882 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:01.074647 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.060501 1066 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:01.060405 32464 raft_consensus.cc:1273] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20250831 07:59:01.075263 31876 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:01.075472 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.076547 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:01.079030 1084 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:01.071369 31880 raft_consensus.cc:2391] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:01.045063 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:01.079566 1084 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.080351 995 raft_consensus.cc:2747] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:01.080348 32469 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:01.081750 1084 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:01.084064 31725 raft_consensus.cc:1238] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:01.084245 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:01.084821 31876 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:01.085148 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:01.085857 32465 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:01.088301 1061 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:01.088773 1061 raft_consensus.cc:695] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 6 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:01.089362 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:01.089569 1061 consensus_queue.cc:237] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.090818 31808 raft_consensus.cc:1273] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:01.091495 951 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:01.110682 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:01.129976 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:01.077574 1066 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:01.094115 32464 pending_rounds.cc:85] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: Aborting all ops after (but not including) 4
I20250831 07:59:01.156879 31847 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:01.065485 909 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:01.073022 1072 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.157806 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.187237 31880 raft_consensus.cc:2391] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:01.159782 942 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:01.161717 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:01.190500 31802 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:01.070657 939 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:01.161537 914 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.162441 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:01.165592 1010 raft_consensus.cc:3053] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:59:01.165249 951 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.212692 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.166813 898 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:01.188737 983 raft_consensus.cc:3053] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:01.168396 31698 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:01.172048 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:01.219635 31809 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:01.173691 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 80309.5us
I20250831 07:59:01.221597 32436 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:01.175736 31879 raft_consensus.cc:1238] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:01.177528 31882 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:01.177785 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:01.178212 31773 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:01.182956 31728 raft_consensus.cc:1273] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:59:01.182901 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:01.184198 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:01.184166 31876 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:01.225468 31733 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:01.191789 31878 raft_consensus.cc:1238] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:01.193804 31725 raft_consensus.cc:1238] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:01.194727 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:01.226397 31808 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:01.189013 953 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:01.204501 31807 raft_consensus.cc:2408] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:01.178457 974 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20250831 07:59:01.176026 1008 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:01.188113 32464 pending_rounds.cc:107] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20250831 07:59:01.166666 31730 raft_consensus.cc:1273] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250831 07:59:01.238437 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 70125.7us
I20250831 07:59:01.206151 1010 raft_consensus.cc:738] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:01.215927 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:01.213654 31877 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:01.184309 1066 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:01.155728 31727 raft_consensus.cc:1273] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:01.192811 909 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.257555 909 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:01.227979 31699 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:01.258385 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:01.259645 31731 raft_consensus.cc:1273] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:59:01.260869 31699 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:01.261624 929 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:01.199437 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.262028 929 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:01.262421 929 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:01.262466 31880 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:01.185422 31726 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:01.265134 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:01.237993 1044 consensus_queue.cc:1046] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
I20250831 07:59:01.242108 978 raft_consensus.cc:2802] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:01.224988 31804 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:01.266613 978 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:01.236510 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:01.266981 978 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:01.267205 31876 raft_consensus.cc:2374] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3. Re-sending same reply.
I20250831 07:59:01.207227 995 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:01.268138 995 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:01.268520 995 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:01.238088 32464 raft_consensus.cc:2887] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:01.269896 951 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:01.245522 1008 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.270598 951 raft_consensus.cc:2747] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:01.236515 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:01.272101 31881 raft_consensus.cc:2374] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2. Re-sending same reply.
I20250831 07:59:01.272281 1008 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:01.217519 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:01.190832 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:01.272897 31875 raft_consensus.cc:1238] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:01.273101 31800 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:01.273252 938 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:59:01.249356 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:01.274672 31807 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:59:01.276150 1044 raft_consensus.cc:3053] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:01.276537 1044 raft_consensus.cc:738] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:01.277992 31803 raft_consensus.cc:1273] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:01.277410 1044 consensus_queue.cc:260] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.278613 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.279228 1044 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:01.280989 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:01.282361 31725 raft_consensus.cc:1273] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:01.283192 31800 raft_consensus.cc:1273] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250831 07:59:01.232429 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 54304.9us
I20250831 07:59:01.289857 31773 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
W20250831 07:59:01.250943 1080 log.cc:927] Time spent T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.242s user 0.000s sys 0.001s
I20250831 07:59:01.251008 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:01.208065 1002 consensus_queue.cc:1046] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.032s
I20250831 07:59:01.291936 1002 consensus_queue.cc:1230] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer a0d845d01c8c4894895f66e3beae298c log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250831 07:59:01.251334 31727 raft_consensus.cc:1238] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:01.292454 1010 consensus_queue.cc:260] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.235664 953 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.293699 1010 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:59:01.293752 939 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:01.294428 939 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:01.294703 953 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:01.234369 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:01.295395 31728 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:59:01.252853 914 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:01.219031 983 raft_consensus.cc:738] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:01.296803 983 consensus_queue.cc:260] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.261135 31731 raft_consensus.cc:1273] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:01.299176 32434 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:01.265069 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:01.230180 31734 raft_consensus.cc:2391] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:01.206633 1074 consensus_queue.cc:1035] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20250831 07:59:01.217835 993 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.027s
I20250831 07:59:01.254822 32465 raft_consensus.cc:2391] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:01.292491 1002 raft_consensus.cc:3053] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:01.265020 31726 raft_consensus.cc:1238] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:01.266160 31805 raft_consensus.cc:1273] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:01.252351 1072 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:01.246910 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.300033 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:01.314460 1067 consensus_queue.cc:1035] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:59:01.302879 31808 raft_consensus.cc:1273] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 07:59:01.307727 919 consensus_queue.cc:1046] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20250831 07:59:01.315881 1002 raft_consensus.cc:738] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:01.316603 1018 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:01.317028 1032 raft_consensus.cc:3053] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:01.317477 1032 raft_consensus.cc:738] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:01.317144 1018 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.318145 1005 consensus_queue.cc:1035] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:59:01.318384 1032 consensus_queue.cc:260] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.319324 1032 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:01.240362 997 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:01.307117 1039 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:01.320434 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:01.320638 1039 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:01.308043 890 consensus_queue.cc:1035] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20250831 07:59:01.321266 1039 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:01.317286 1002 consensus_queue.cc:260] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.311615 1011 consensus_queue.cc:1046] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20250831 07:59:01.322286 1002 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:01.322590 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:01.323230 32463 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:01.316154 1067 consensus_queue.cc:1035] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:59:01.301702 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:01.324880 31804 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:01.327307 31773 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
W20250831 07:59:01.273636 907 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20250831 07:59:01.298390 983 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:01.312395 1023 raft_consensus.cc:2747] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:01.330763 1005 consensus_queue.cc:1035] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:59:01.332037 1011 raft_consensus.cc:3053] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:01.332425 1011 raft_consensus.cc:738] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:01.289728 1013 raft_consensus.cc:2953] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:59:01.333150 1011 consensus_queue.cc:260] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.334293 1011 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:01.338850 919 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:01.339365 919 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:01.339790 919 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:01.352145 31849 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:01.353049 31699 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:01.341856 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:01.354126 31808 raft_consensus.cc:2391] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:01.354291 31880 raft_consensus.cc:1273] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:01.354929 31808 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:39082
I20250831 07:59:01.341398 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:01.355446 31808 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:59:01.355674 31800 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:59:01.355849 31808 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:59:01.357339 31877 raft_consensus.cc:1273] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:01.237886 1036 raft_consensus.cc:2802] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:01.358561 31877 raft_consensus.cc:1273] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:59:01.358642 1036 raft_consensus.cc:695] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:01.341053 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:01.340106 909 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:01.359632 1036 consensus_queue.cc:237] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.359977 909 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:01.360577 909 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:01.344587 31801 raft_consensus.cc:1273] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250831 07:59:01.345211 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.345131 940 consensus_queue.cc:1046] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:59:01.361898 940 raft_consensus.cc:3053] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:01.346081 1067 consensus_queue.cc:1035] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:59:01.354169 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:01.359514 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:01.340940 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:01.349555 1008 consensus_queue.cc:1046] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:59:01.352160 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.353051 31849 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:01.380060 31806 raft_consensus.cc:2391] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:01.346513 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.362183 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.362787 31801 pending_rounds.cc:85] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 1
I20250831 07:59:01.315656 32434 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:01.381268 31802 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:01.381512 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:01.381798 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 66210.4us
I20250831 07:59:01.382332 31882 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:01.382520 31802 raft_consensus.cc:1273] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:01.383452 32434 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:01.386291 32434 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:01.366287 31728 raft_consensus.cc:1273] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:01.388229 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:01.349050 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 104634us
I20250831 07:59:01.389221 31882 raft_consensus.cc:2391] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:01.389231 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:01.373587 32435 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:01.362037 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:01.345034 1018 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:01.361034 954 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:01.384528 31699 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:01.384861 31805 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:01.381299 31801 pending_rounds.cc:107] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250831 07:59:01.399915 31801 raft_consensus.cc:2887] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:01.320593 997 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.395175 31848 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:01.378787 31809 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:01.401552 31800 raft_consensus.cc:1238] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 07:59:01.358727 993 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:01.403373 31699 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:01.403555 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.404445 31882 raft_consensus.cc:2391] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:01.404398 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:01.407856 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:01.411310 31875 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:01.362756 940 raft_consensus.cc:738] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:01.383298 1074 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:01.387669 1072 raft_consensus.cc:2802] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 07:59:01.451825 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:01.428511 31698 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:01.387378 953 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:01.410296 917 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:01.393719 934 raft_consensus.cc:2802] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:01.505587 31875 raft_consensus.cc:1238] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:01.384011 998 consensus_queue.cc:1035] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.103s
I20250831 07:59:01.509382 31876 raft_consensus.cc:1273] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:59:01.509931 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:01.510923 31878 raft_consensus.cc:2391] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:01.394585 1023 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:59:01.387370 890 consensus_queue.cc:1035] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.106s
I20250831 07:59:01.409052 938 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:01.482553 1084 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:01.530838 31732 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
W20250831 07:59:01.508103 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 79748.6us
I20250831 07:59:01.394675 907 raft_consensus.cc:2802] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:59:01.514681 31879 raft_consensus.cc:1273] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250831 07:59:01.490815 1072 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:01.412909 1007 consensus_queue.cc:1035] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.102s
I20250831 07:59:01.393121 941 consensus_queue.cc:1035] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.113s
I20250831 07:59:01.516191 31875 raft_consensus.cc:1238] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:01.402932 985 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:01.509400 934 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:01.398149 1001 consensus_queue.cc:1035] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.116s
I20250831 07:59:01.387087 978 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.487496 1018 consensus_queue.cc:1035] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20250831 07:59:01.410324 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:01.556596 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 335078us
I20250831 07:59:01.393982 980 consensus_queue.cc:1035] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.117s
I20250831 07:59:01.481614 1085 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:01.621506 31804 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:01.575095 31879 pending_rounds.cc:85] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10: Aborting all ops after (but not including) 1
I20250831 07:59:01.667541 31879 pending_rounds.cc:107] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250831 07:59:01.667932 31879 raft_consensus.cc:2887] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:01.505839 914 raft_consensus.cc:2802] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:01.576399 31698 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:01.609525 1072 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:01.514302 917 raft_consensus.cc:695] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:01.674194 917 consensus_queue.cc:237] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:01.675009 997 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:01.524374 908 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:01.675618 997 consensus_queue.cc:1035] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:59:01.642833 31878 raft_consensus.cc:1273] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:59:01.530086 938 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:01.678134 31808 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.678695 938 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:01.554461 940 consensus_queue.cc:260] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.669679 992 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:01.680423 940 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:01.680285 992 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.413493 954 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.627698 907 raft_consensus.cc:695] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:01.626367 31876 raft_consensus.cc:1238] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
W20250831 07:59:01.643110 31628 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.007s sys 0.058s
W20250831 07:59:01.755805 31628 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.695s user 0.007s sys 0.058s
I20250831 07:59:01.494796 1074 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.555716 898 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:01.626305 934 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:01.506135 953 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:01.645226 1086 log.cc:927] Time spent T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20250831 07:59:01.683588 31877 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:59:01.492332 929 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.635807 951 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:01.708091 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:01.755183 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 370199us
I20250831 07:59:01.757283 31808 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:01.639843 939 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.794222 31882 raft_consensus.cc:1273] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 07:59:01.625021 1084 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.806716 31773 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:01.757604 913 raft_consensus.cc:2802] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:01.810724 31773 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:01.818032 31733 raft_consensus.cc:1238] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:01.833375 1084 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:01.790534 983 consensus_queue.cc:1035] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20250831 07:59:01.792894 898 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.810298 951 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.808549 909 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.835902 993 raft_consensus.cc:695] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
W20250831 07:59:01.857870 919 consensus_meta.cc:287] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.507s user 0.009s sys 0.000s
I20250831 07:59:01.881743 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:01.897080 31808 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:01.852866 1007 raft_consensus.cc:3053] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:01.881665 908 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:01.880419 998 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:01.880263 1084 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:01.928001 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:01.928851 31804 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:59:01.905707 1007 raft_consensus.cc:738] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:01.926124 1092 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:01.938936 1092 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.941124 1092 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:01.942210 32463 raft_consensus.cc:2953] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:59:01.947409 31808 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:01.948423 1007 consensus_queue.cc:260] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.950394 1007 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:59:01.987516 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:02.025553 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:01.915243 1090 log.cc:927] Time spent T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
W20250831 07:59:02.031080 31847 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.009s sys 0.015s
I20250831 07:59:01.878132 907 consensus_queue.cc:237] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:01.930972 978 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:01.910521 919 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:02.039969 31847 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s user 0.009s sys 0.015s
W20250831 07:59:02.035907 31772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.009s sys 0.016s
W20250831 07:59:02.040463 31772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.655s user 0.009s sys 0.016s
W20250831 07:59:02.026167 1044 consensus_meta.cc:287] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.747s user 0.012s sys 0.000s
I20250831 07:59:02.045543 1100 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:02.047320 1097 log.cc:927] Time spent T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
I20250831 07:59:01.930637 908 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:01.935047 998 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:02.057391 31881 raft_consensus.cc:2408] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250831 07:59:02.040066 1013 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.002s sys 0.022s
I20250831 07:59:02.033982 954 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:02.084046 31808 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:02.087510 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:02.088310 31804 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:01.956894 1084 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:02.049630 1032 consensus_meta.cc:287] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.730s user 0.007s sys 0.005s
W20250831 07:59:02.093360 1093 log.cc:927] Time spent T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
W20250831 07:59:02.094286 1094 log.cc:927] Time spent T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.213s user 0.001s sys 0.000s
I20250831 07:59:02.094826 1060 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:59:02.057608 939 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:02.096860 1098 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:02.101148 919 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:02.113813 890 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:02.115163 1095 log.cc:927] Time spent T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.218s user 0.000s sys 0.001s
W20250831 07:59:02.116307 1096 log.cc:927] Time spent T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20250831 07:59:02.040999 953 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:02.057489 929 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:02.135473 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:02.082885 1013 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s user 0.002s sys 0.023s
I20250831 07:59:02.153467 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:02.127157 1066 consensus_meta.cc:287] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.874s user 0.010s sys 0.000s
W20250831 07:59:02.157754 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 581476us
I20250831 07:59:02.160753 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:59:02.158447 939 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:02.194443 917 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:02.180666 31804 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:02.133498 908 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:02.137671 954 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:02.168341 31849 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.009s sys 0.022s
I20250831 07:59:02.152267 1032 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:02.176066 31848 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.006s sys 0.027s
I20250831 07:59:02.159766 914 raft_consensus.cc:695] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:02.131040 1005 raft_consensus.cc:2747] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:02.135540 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:02.231387 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:02.232273 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:02.232823 31880 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:59:02.194698 1036 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.826s user 0.013s sys 0.025s
W20250831 07:59:02.219695 31849 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.737s user 0.009s sys 0.022s
W20250831 07:59:02.237630 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 832220us
W20250831 07:59:02.224951 31848 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.824s user 0.006s sys 0.027s
W20250831 07:59:02.238978 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 843797us
I20250831 07:59:02.204376 951 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:02.227226 32435 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:02.234649 31698 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:02.231508 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:02.203114 953 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:02.053423 993 consensus_queue.cc:237] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.078917 1100 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.256965 31876 raft_consensus.cc:1273] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:02.225172 978 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:02.265105 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:02.249908 31882 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
W20250831 07:59:02.254560 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:02.256289 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 118) took 1070 ms. Trace:
W20250831 07:59:02.237422 1036 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.875s user 0.014s sys 0.025s
I20250831 07:59:02.173961 890 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.259203 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1106) took 1014 ms. Trace:
I20250831 07:59:02.244081 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:02.432334 31809 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:02.205041 934 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.305461 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:02.234887 1013 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57332 (request call id 590) took 2394 ms. Trace:
I20250831 07:59:02.437847 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1110) took 1078 ms. Trace:
W20250831 07:59:02.430907 31773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.007s sys 0.026s
I20250831 07:59:02.209127 1066 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:02.228364 1032 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.228829 954 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.256567 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:02.228029 917 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:02.341899 898 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:02.359995 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:02.256985 953 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.459345 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 117) took 1283 ms. Trace:
I20250831 07:59:02.224071 1085 raft_consensus.cc:695] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:02.220180 913 raft_consensus.cc:695] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:02.420291 31808 raft_consensus.cc:1238] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
W20250831 07:59:02.420341 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 135206us
I20250831 07:59:02.515995 31730 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:02.362800 1107 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:02.424614 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 160561us
I20250831 07:59:02.425794 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:02.491742 995 consensus_meta.cc:287] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.223s user 0.016s sys 0.000s
I20250831 07:59:02.427559 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:02.456512 1002 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.961s user 0.013s sys 0.045s
I20250831 07:59:02.430836 31729 rpcz_store.cc:276] 0831 07:59:01.245064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.266638 (+ 21574us) service_pool.cc:225] Handling call
0831 07:59:01.266975 (+ 337us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:01.267373 (+ 398us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:01.267409 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:01.267977 (+ 568us) log.cc:844] Serialized 26 byte log entry
0831 07:59:01.609537 (+341560us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:01.609647 (+ 110us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:01.609752 (+ 105us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:02.258943 (+649191us) raft_consensus.cc:1683] finished
0831 07:59:02.258971 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:02.259132 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4064,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":124,"thread_start_us":341119,"threads_started":1,"wal-append.queue_time_us":568515}
I20250831 07:59:02.435739 998 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:02.220110 939 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:02.440790 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:02.454701 31773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s user 0.012s sys 0.027s
W20250831 07:59:02.455750 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:02.442523 31734 rpcz_store.cc:276] 0831 07:59:01.347666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.347839 (+ 173us) service_pool.cc:225] Handling call
0831 07:59:01.348132 (+ 293us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:01.348460 (+ 328us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:01.348491 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:01.348979 (+ 488us) log.cc:844] Serialized 26 byte log entry
0831 07:59:01.791350 (+442371us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:01.791434 (+ 84us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:01.791704 (+ 270us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:02.426175 (+634471us) raft_consensus.cc:1683] finished
0831 07:59:02.426205 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:02.426364 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3909,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":127,"thread_start_us":441846,"threads_started":1,"wal-append.queue_time_us":570033}
I20250831 07:59:02.257493 1092 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:02.495409 31874 rpcz_store.cc:276] 0831 07:59:01.175493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.176510 (+ 1017us) service_pool.cc:225] Handling call
0831 07:59:01.182671 (+ 6161us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:01.197990 (+ 15319us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:01.198045 (+ 55us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:01.217416 (+ 19371us) log.cc:844] Serialized 26 byte log entry
0831 07:59:01.628015 (+410599us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:01.666583 (+ 38568us) log.cc:844] Serialized 14 byte log entry
0831 07:59:01.666771 (+ 188us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:01.666867 (+ 96us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:02.459073 (+792206us) raft_consensus.cc:1683] finished
0831 07:59:02.459104 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:02.459276 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7622,"prepare.queue_time_us":38388,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":23369,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":12264,"thread_start_us":371936,"threads_started":1,"wal-append.queue_time_us":577330}
I20250831 07:59:02.520870 31877 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:02.426539 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:02.455781 1104 log.cc:927] Time spent T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.378s user 0.002s sys 0.000s
I20250831 07:59:02.524785 31725 raft_consensus.cc:2391] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:02.525365 31882 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:02.458613 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:02.423195 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:02.552409 31806 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:02.552860 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:02.553519 31876 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:59:02.426533 978 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.438772 909 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:02.556238 909 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:02.556741 909 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:02.557394 978 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:02.403162 914 consensus_queue.cc:237] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.422137 31883 rpcz_store.cc:276] 0831 07:59:01.176787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.177084 (+ 297us) service_pool.cc:225] Handling call
0831 07:59:01.191227 (+ 14143us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:01.205145 (+ 13918us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:01.205177 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:01.218823 (+ 13646us) log.cc:844] Serialized 26 byte log entry
0831 07:59:01.284859 (+ 66036us) spinlock_profiling.cc:243] Waited 7.27 ms on lock 0x7b18003007c0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b792 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:59:01.669428 (+384569us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:01.669472 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:01.669537 (+ 65us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:02.228725 (+559188us) raft_consensus.cc:1683] finished
0831 07:59:02.228748 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:02.247076 (+ 18328us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13548,"raft.queue_time_us":13828,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":175,"spinlock_wait_cycles":17016448,"thread_start_us":433592,"threads_started":1,"wal-append.queue_time_us":677576}
I20250831 07:59:02.434648 1110 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:59:02.544100 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 733446us
I20250831 07:59:02.571600 31773 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:02.547318 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 122849us
I20250831 07:59:02.561453 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:02.572746 31729 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:02.547756 31805 raft_consensus.cc:2953] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:59:02.575567 31773 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:02.576948 31773 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:02.578640 31848 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:02.541733 1002 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.043s user 0.013s sys 0.047s
I20250831 07:59:02.580693 32471 raft_consensus.cc:2391] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:02.582433 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:02.582892 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:02.583786 31877 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:59:02.589107 31882 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:02.589823 31773 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:02.542502 1101 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:02.594126 1121 log.cc:927] Time spent T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20250831 07:59:02.594743 1101 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.532893 995 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:02.613009 995 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:02.536975 908 consensus_queue.cc:1046] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.044s
I20250831 07:59:02.613693 1092 raft_consensus.cc:695] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:02.607421 909 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:02.584080 919 raft_consensus.cc:2747] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:02.615089 919 raft_consensus.cc:2747] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:02.614696 1085 consensus_queue.cc:237] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.614785 1092 consensus_queue.cc:237] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:02.583896 1036 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:59:02.596712 31732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1555) took 1626 ms (client timeout 1579 ms). Trace:
W20250831 07:59:02.616099 31732 rpcz_store.cc:269] 0831 07:59:00.950743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:00.951309 (+ 566us) service_pool.cc:225] Handling call
0831 07:59:02.577143 (+1625834us) spinlock_profiling.cc:243] Waited 543 ms on lock 0x7b5400091854. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:02.577244 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1249006848}
W20250831 07:59:02.617045 938 consensus_meta.cc:287] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.938s user 0.009s sys 0.001s
I20250831 07:59:02.617442 938 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:02.582854 1039 consensus_meta.cc:287] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 1.261s user 0.012s sys 0.000s
I20250831 07:59:02.560825 1084 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:02.630780 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 109062us
I20250831 07:59:02.631441 31848 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:02.637519 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 116837us
I20250831 07:59:02.533576 1107 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.651172 31806 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:02.653671 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:02.627590 919 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:02.628166 938 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:02.658555 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:02.669240 31808 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 4
I20250831 07:59:02.619870 1039 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:02.663312 31880 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:02.658623 890 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:02.673913 1119 consensus_queue.cc:1035] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.148s
I20250831 07:59:02.666612 909 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:02.653362 995 raft_consensus.cc:2802] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election won for term 3
W20250831 07:59:02.680917 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 91178.8us
I20250831 07:59:02.667807 1100 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:02.675611 890 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:02.683640 1130 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:02.684381 953 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:02.596203 913 consensus_queue.cc:237] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.692723 31773 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:02.688138 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:02.640854 1106 consensus_queue.cc:1035] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20250831 07:59:02.692054 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:02.703296 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:02.683496 1111 log.cc:927] Time spent T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.192s user 0.000s sys 0.001s
I20250831 07:59:02.703495 31802 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:02.704087 31876 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:02.700783 31880 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:02.591734 1110 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.685798 908 raft_consensus.cc:3053] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:02.691329 1032 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:02.699337 1120 log.cc:927] Time spent T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.061s user 0.001s sys 0.001s
I20250831 07:59:02.694386 1074 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:02.741341 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:02.740732 31773 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:02.695516 1130 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:02.690935 934 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:02.745987 31880 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:02.697209 1066 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:02.696851 917 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:02.743279 31808 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:02.700153 954 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:02.702926 1100 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:02.702548 995 raft_consensus.cc:695] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
W20250831 07:59:02.738756 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 63683.5us
I20250831 07:59:02.682607 909 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:02.740396 1010 consensus_meta.cc:287] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.446s user 0.012s sys 0.002s
I20250831 07:59:02.754122 32435 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:02.717176 908 raft_consensus.cc:738] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:02.765759 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:02.761193 31802 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:02.741410 907 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:02.713519 890 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.760682 31876 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:02.755096 1018 raft_consensus.cc:2747] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:02.784143 31880 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:02.770390 1110 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:02.748013 1130 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:02.769661 934 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:02.791234 31880 raft_consensus.cc:604] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:02.796371 31880 raft_consensus.cc:612] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:59:02.770208 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:02.798364 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:02.798990 32464 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:02.442365 1013 rpcz_store.cc:276] 0831 07:58:59.817502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:59.817652 (+ 150us) service_pool.cc:225] Handling call
0831 07:58:59.942038 (+124386us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b540019b520. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7f91c50 00007fdce8003af2 00007fdce80027b0 00007fdce7ffb9ba 00007fdcecf5b079 00007fdce65eb644 00007fdce65eb5d2 00007fdce65eb501 00007fdce65eb47c 00007fdce65ea732
0831 07:58:59.976423 (+ 34385us) log.cc:844] Serialized 494 byte log entry
0831 07:59:00.588738 (+612315us) spinlock_profiling.cc:243] Waited 4.32 ms on lock 0x7b540019b520. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7f95256 00007fdce7f8ebca 00007fdce7f8de34 00007fdce7f76502 00007fdce7f7b911 00007fdce7f7b899 00007fdce7f7b829 00007fdce7f7b7f1 00007fdce7f7a9bd 0000000000379254
0831 07:59:02.209281 (+1620543us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51280,"raft.queue_time_us":17088,"raft.run_cpu_time_us":11966,"raft.run_wall_time_us":452203,"spinlock_wait_cycles":60865024,"thread_start_us":162150,"threads_started":1,"wal-append.queue_time_us":262027,"wal-append.run_cpu_time_us":1648,"wal-append.run_wall_time_us":1010642}
I20250831 07:59:02.780025 1098 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:02.784253 939 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:02.786921 1100 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:02.785800 938 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:02.775712 1032 consensus_queue.cc:1035] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20250831 07:59:02.792261 32465 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:59:02.796767 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:02.761587 1138 consensus_queue.cc:1035] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20250831 07:59:02.805752 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:02.804170 32434 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:02.821467 31802 raft_consensus.cc:2466] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:59:02.806916 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:02.808135 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:02.822724 31880 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:02.823856 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:02.808756 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:02.810832 31848 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:02.812389 1108 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:02.799500 1067 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:02.829236 1067 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:02.831364 1113 log.cc:927] Time spent T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20250831 07:59:02.823902 992 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:02.792773 907 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:02.815929 995 consensus_queue.cc:237] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:02.801740 934 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:02.861394 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 50637.2us
I20250831 07:59:02.864812 32434 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:02.865482 31848 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:02.828059 1108 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:02.828411 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:02.832463 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1113) took 1467 ms. Trace:
I20250831 07:59:02.835757 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:02.875142 31880 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:02.875895 31880 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:02.802451 1101 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:02.804010 1107 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:02.862829 31773 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:02.862844 1011 consensus_meta.cc:287] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 1.528s user 0.014s sys 0.001s
I20250831 07:59:02.868484 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:02.883198 31773 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:02.873953 31728 rpcz_store.cc:276] 0831 07:59:01.365410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.506448 (+141038us) service_pool.cc:225] Handling call
0831 07:59:01.506871 (+ 423us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:01.627874 (+121003us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:01.627924 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:01.628526 (+ 602us) log.cc:844] Serialized 26 byte log entry
0831 07:59:02.232476 (+603950us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:02.232505 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:02.232549 (+ 44us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:02.832215 (+599666us) raft_consensus.cc:1683] finished
0831 07:59:02.832241 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:02.832395 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25729,"raft.queue_time_us":25976,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":74127,"thread_start_us":476879,"threads_started":1,"wal-append.queue_time_us":997665}
I20250831 07:59:02.835042 929 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:02.835142 938 raft_consensus.cc:695] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:02.818974 1100 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:02.872164 940 consensus_meta.cc:287] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 1.191s user 0.008s sys 0.005s
I20250831 07:59:02.864698 992 raft_consensus.cc:2802] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:02.864787 939 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:02.893564 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:02.862767 1106 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:02.831369 1067 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:02.871541 1039 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:02.901546 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:02.807075 908 consensus_queue.cc:260] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:02.820580 1098 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:02.888252 1023 consensus_meta.cc:287] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 1.300s user 0.014s sys 0.000s
I20250831 07:59:02.881518 1013 raft_consensus.cc:2802] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:02.885628 1136 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:02.886263 1138 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:02.894454 1101 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:02.956269 31881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1109) took 1609 ms (client timeout 1643 ms). Trace:
I20250831 07:59:02.912243 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:02.939844 31802 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:02.921463 31804 raft_consensus.cc:1273] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250831 07:59:03.008737 31804 pending_rounds.cc:85] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 1
I20250831 07:59:03.009056 31804 pending_rounds.cc:107] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250831 07:59:03.009347 31804 raft_consensus.cc:2887] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250831 07:59:02.915127 1091 log.cc:927] Time spent T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 1.385s user 0.000s sys 0.004s
I20250831 07:59:02.893785 1011 raft_consensus.cc:2762] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20250831 07:59:03.017221 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 782645us
I20250831 07:59:02.980628 31847 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:02.919972 31616 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 26525016efb247c3b1c2bf7fec0e1658 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 07:59:03.017578 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:03.018103 31616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 151712us
I20250831 07:59:03.018409 31732 raft_consensus.cc:2408] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:03.010948 32435 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:03.002825 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:03.019340 31876 raft_consensus.cc:2391] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:02.912550 1133 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:02.974355 1093 log.cc:927] Time spent T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20250831 07:59:02.979084 1032 raft_consensus.cc:2802] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 07:59:03.006646 31881 rpcz_store.cc:269] 0831 07:59:01.347224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.351926 (+ 4702us) service_pool.cc:225] Handling call
0831 07:59:02.955952 (+1604026us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b54002bc854. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:02.956164 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1555208448}
I20250831 07:59:02.918040 978 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.012401 992 raft_consensus.cc:695] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:03.016242 907 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:02.901798 929 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.083787 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.084443 31732 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
W20250831 07:59:02.912817 1125 log.cc:927] Time spent T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.156s user 0.000s sys 0.002s
I20250831 07:59:02.917836 1116 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:02.913506 1072 consensus_meta.cc:287] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 1.156s user 0.007s sys 0.005s
I20250831 07:59:03.013172 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.091629 31728 raft_consensus.cc:2408] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 07:59:02.994994 1136 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:02.978829 1013 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:02.979158 1023 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:02.995214 939 raft_consensus.cc:695] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:03.089639 1091 log.cc:927] Time spent T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
W20250831 07:59:03.062294 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:02.995836 983 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.030459 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:03.030488 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 49999.1us
I20250831 07:59:03.075368 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:03.078775 31698 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:03.117139 942 raft_consensus.cc:2802] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:03.117724 942 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:03.118057 1146 log.cc:927] Time spent T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.196s user 0.000s sys 0.000s
I20250831 07:59:03.118338 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:03.119872 32470 raft_consensus.cc:1238] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:59:02.973170 951 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:03.081627 1061 consensus_queue.cc:1035] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 07:59:03.072799 1137 log.cc:927] Time spent T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.296s user 0.001s sys 0.000s
I20250831 07:59:03.075183 1147 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.103407 1013 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:03.088833 31699 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:02.899058 1134 consensus_queue.cc:1046] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.314s
I20250831 07:59:03.067560 954 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.091624 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1115) took 1735 ms. Trace:
I20250831 07:59:03.028990 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.023849 890 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.025144 909 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.088644 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:03.085438 978 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.012990 1138 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.074677 1010 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:03.095067 1123 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:03.098342 1148 log.cc:927] Time spent T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20250831 07:59:03.095388 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:03.076377 898 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:03.031906 1133 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.105636 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1568) took 1741 ms. Trace:
I20250831 07:59:03.142366 1066 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:03.016695 1126 consensus_queue.cc:1035] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.175s
I20250831 07:59:03.171479 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:03.172683 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:57332 (request call id 591) took 1804 ms. Trace:
I20250831 07:59:03.173074 31875 rpcz_store.cc:276] 0831 07:59:01.367874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.629528 (+261654us) service_pool.cc:225] Handling call
0831 07:59:03.172245 (+1542717us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400410854. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f81cd0 00007fdce8005eea 00007fdcecf5e60e 00007fdce65fbcc7 00007fdce65fbc52 00007fdce65fbb81 00007fdce65fbafc 00007fdce65fadb2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:03.172607 (+ 362us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2622687616}
I20250831 07:59:03.174618 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:03.117023 31698 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:03.082485 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:03.180351 31876 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:03.080667 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:02.917394 1114 raft_consensus.cc:3058] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:59:03.021811 908 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:03.018668 1101 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.130919 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1435) took 1757 ms. Trace:
I20250831 07:59:03.121299 983 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.131903 929 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.185039 1133 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:03.196110 31725 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:03.131996 951 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.198573 951 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:03.132645 939 consensus_queue.cc:237] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.201696 1114 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:03.186249 1161 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:03.134509 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.204121 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.202972 1161 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:03.204804 31732 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:03.184793 1066 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.100418 1136 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:03.139155 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 1.20766e+06us
W20250831 07:59:03.117185 1132 log.cc:927] Time spent T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.087s user 0.001s sys 0.001s
I20250831 07:59:03.140489 31879 rpcz_store.cc:276] 0831 07:59:01.356515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.356622 (+ 107us) service_pool.cc:225] Handling call
0831 07:59:01.356844 (+ 222us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:01.668610 (+311766us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:03.091410 (+1422800us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54001ecdd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:03.091567 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":318916864}
I20250831 07:59:03.118119 942 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:03.117257 1110 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.099021 1072 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.223654 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:03.224481 31879 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:59:03.225579 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:03.226392 31879 raft_consensus.cc:2366] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4.
I20250831 07:59:03.227205 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:03.153527 32468 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:03.078339 938 consensus_queue.cc:237] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.077574 1032 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:02.974788 1106 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.143887 934 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:03.145573 1085 raft_consensus.cc:2802] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:03.164369 31775 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:03.163328 31727 rpcz_store.cc:276] 0831 07:59:01.364651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.365010 (+ 359us) service_pool.cc:225] Handling call
0831 07:59:01.365281 (+ 271us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 07:59:01.384355 (+ 19074us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:01.384385 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 07:59:01.410755 (+ 26370us) log.cc:844] Serialized 52 byte log entry
0831 07:59:01.758568 (+347813us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:01.814887 (+ 56319us) log.cc:844] Serialized 14 byte log entry
0831 07:59:02.229572 (+414685us) log.cc:844] Serialized 14 byte log entry
0831 07:59:02.832392 (+602820us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b18002fd100. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce501365a 00007fdcee4e4409 00007fdce4ed4af0 00007fdce5098894 00007fdce5098805 00007fdce7ff5020 00007fdce80076c6 00007fdce8018237 00007fdce80181b8
0831 07:59:03.105113 (+272721us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:03.105217 (+ 104us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:03.105376 (+ 159us) raft_consensus.cc:1683] finished
0831 07:59:03.105397 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:03.105566 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24569,"prepare.queue_time_us":61262,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":77,"raft.queue_time_us":732582,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":9798,"spinlock_wait_cycles":24924032,"thread_start_us":950674,"threads_started":3,"wal-append.queue_time_us":442354}
I20250831 07:59:03.176347 978 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:03.109527 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:03.242942 1085 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:03.246726 1085 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:03.148878 1147 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.249228 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1544) took 1007 ms. Trace:
I20250831 07:59:03.249619 31804 rpcz_store.cc:276] 0831 07:59:02.241912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:02.244278 (+ 2366us) service_pool.cc:225] Handling call
0831 07:59:02.244606 (+ 328us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:03.009905 (+765299us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:03.248969 (+239064us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b5400358c54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:03.249164 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1555727232}
I20250831 07:59:03.182844 1098 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:03.182930 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 66016.7us
I20250831 07:59:03.252036 31698 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:03.243538 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 79263.9us
I20250831 07:59:03.253652 31775 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:03.254037 31698 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:03.185525 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 174229us
I20250831 07:59:03.179847 31875 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:03.119314 992 consensus_queue.cc:237] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.256183 1011 raft_consensus.cc:2802] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:03.256594 1011 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:03.256955 1011 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:03.257241 1061 raft_consensus.cc:2802] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:03.190263 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:03.257779 1061 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:03.258283 1061 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:03.146489 954 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.260108 978 raft_consensus.cc:2802] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:03.260783 978 raft_consensus.cc:695] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:03.261402 954 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:03.090870 1116 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:03.185822 1007 consensus_meta.cc:287] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 1.228s user 0.014s sys 0.000s
I20250831 07:59:03.190887 31807 rpcz_store.cc:276] 0831 07:59:01.367235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.367394 (+ 159us) service_pool.cc:225] Handling call
0831 07:59:01.392231 (+ 24837us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:01.858082 (+465851us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:01.858115 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:02.684664 (+826549us) spinlock_profiling.cc:243] Waited 46.7 ms on lock 0x7b2c00000c88. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce69a09a1 00007fdce69affd2 00007fdce8039021 00007fdce7ffd3fa 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32
0831 07:59:02.685060 (+ 396us) log.cc:844] Serialized 26 byte log entry
0831 07:59:02.755323 (+ 70263us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:02.755362 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:02.755429 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:03.119705 (+364276us) raft_consensus.cc:1683] finished
0831 07:59:03.119729 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:03.124531 (+ 4802us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1003,"raft.queue_time_us":478487,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":6261,"spinlock_wait_cycles":107449600,"thread_start_us":517252,"threads_started":2,"wal-append.queue_time_us":175005}
I20250831 07:59:03.195261 983 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.166745 898 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.245646 1099 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.273175 1099 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.274164 898 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.275105 1099 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.246264 890 raft_consensus.cc:2747] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:03.230793 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.259673 1008 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.235999 929 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.276564 1008 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.277169 929 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.278497 1008 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.236951 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.157632 1123 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.239933 1032 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:03.268342 1116 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:03.204810 1161 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:03.270650 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.298281 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.263217 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1548) took 1020 ms. Trace:
I20250831 07:59:03.221729 1100 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.280128 31880 raft_consensus.cc:604] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:03.231050 1108 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:03.279045 1138 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.317497 1101 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:03.318885 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.319550 31807 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
W20250831 07:59:03.287045 31730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1562) took 2103 ms (client timeout 2715 ms). Trace:
I20250831 07:59:03.287492 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:03.321544 31732 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:03.321249 31730 rpcz_store.cc:269] 0831 07:59:01.184062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.244791 (+ 60729us) service_pool.cc:225] Handling call
0831 07:59:03.286880 (+2042089us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400021a54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:03.286987 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2816011648}
I20250831 07:59:03.237872 974 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:03.289573 31698 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:03.291903 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.323268 31809 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:03.254740 31775 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:03.223101 951 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.299949 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.302286 31880 raft_consensus.cc:612] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:59:03.327250 31698 leader_election.cc:400] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Vote denied by peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585) with higher term. Message: Invalid argument: T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4.
I20250831 07:59:03.327786 31698 leader_election.cc:403] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250831 07:59:03.330502 31698 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:03.304909 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 364258us
W20250831 07:59:03.306100 1163 log.cc:927] Time spent T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.338s user 0.004s sys 0.000s
I20250831 07:59:03.299656 31805 rpcz_store.cc:276] 0831 07:59:02.243149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:02.243758 (+ 609us) service_pool.cc:225] Handling call
0831 07:59:02.419458 (+175700us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:02.419787 (+ 329us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:02.419843 (+ 56us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:02.436648 (+ 16805us) log.cc:844] Serialized 494 byte log entry
0831 07:59:02.547645 (+110997us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:02.620313 (+ 72668us) log.cc:844] Serialized 14 byte log entry
0831 07:59:02.620585 (+ 272us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:02.620739 (+ 154us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:03.262938 (+642199us) raft_consensus.cc:1683] finished
0831 07:59:03.262967 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:03.263146 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":81,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":12197,"thread_start_us":110625,"threads_started":1,"wal-append.queue_time_us":483986}
I20250831 07:59:03.306412 1130 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.261893 978 consensus_queue.cc:237] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.307500 1123 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:03.337023 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 82366us
I20250831 07:59:03.333223 31773 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:03.338316 31775 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:03.230804 1110 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.310981 1106 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:03.279788 954 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.286460 1147 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:03.303615 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:03.347641 31727 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:03.289160 929 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:03.341461 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.298195 1066 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:03.340639 31698 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:03.342671 32436 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:03.344604 31773 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:03.303381 32465 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:59:03.359675 31881 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:03.358901 31698 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:03.335170 1116 raft_consensus.cc:2802] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:03.361641 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.366772 32468 raft_consensus.cc:1238] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:59:03.355880 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.370786 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.370862 32436 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:03.375561 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.380729 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.381568 31804 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:03.382926 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.383553 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:03.394665 31732 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:03.396790 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1567) took 2032 ms. Trace:
I20250831 07:59:03.397447 31698 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:03.345530 1100 raft_consensus.cc:2747] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:03.400955 31809 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:03.417582 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.397226 31733 rpcz_store.cc:276] 0831 07:59:01.364389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.364602 (+ 213us) service_pool.cc:225] Handling call
0831 07:59:01.364836 (+ 234us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:01.818569 (+453733us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:03.396570 (+1578001us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b54002eb8d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:03.396734 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1041730432}
I20250831 07:59:03.423735 31773 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:03.423498 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.336956 951 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.351701 1138 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:03.426306 31732 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:03.424064 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.439211 31801 raft_consensus.cc:2391] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 4: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:03.418344 31809 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:03.440227 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.372469 1008 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250831 07:59:03.434520 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 63742.2us
I20250831 07:59:03.442149 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.450682 32436 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:03.443069 1067 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:03.443498 31773 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:03.443966 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.444442 1044 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.445437 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 122) took 1778 ms. Trace:
I20250831 07:59:03.445993 1032 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.433070 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.467752 31803 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:03.423674 929 raft_consensus.cc:2747] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:03.369243 983 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:03.468886 31876 raft_consensus.cc:1273] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20250831 07:59:03.469264 31876 pending_rounds.cc:85] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10: Aborting all ops after (but not including) 3
I20250831 07:59:03.469491 31876 pending_rounds.cc:107] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20250831 07:59:03.469735 31876 raft_consensus.cc:2887] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:03.350020 954 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.442476 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.444047 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.453347 31807 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 4: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:03.384120 1007 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.384828 1139 raft_consensus.cc:2747] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:03.458648 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.396046 1114 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585) with higher term. Message: Invalid argument: T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4.
I20250831 07:59:03.479406 954 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:03.479676 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:03.480858 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:03.441241 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.454890 909 raft_consensus.cc:2747] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:03.433825 1066 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:03.423552 1116 raft_consensus.cc:695] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:03.385581 890 consensus_queue.cc:1035] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.092s
I20250831 07:59:03.455855 1008 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:03.477057 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.438268 1138 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:03.477993 1007 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.483083 31699 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:03.497730 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.498312 31807 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:03.498565 31699 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:03.473773 1100 raft_consensus.cc:2802] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 07:59:03.483809 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.506757 31809 raft_consensus.cc:2391] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:03.447746 985 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:03.461400 1137 log.cc:927] Time spent T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250831 07:59:03.480906 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:03.504235 1116 consensus_queue.cc:237] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.453629 1001 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.508390 1001 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.510726 1001 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:03.459759 907 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:03.512215 1007 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.512431 907 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:03.453224 32464 consensus_meta.cc:287] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.654s user 0.005s sys 0.005s
I20250831 07:59:03.493289 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:03.518006 31803 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:03.493629 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.485183 1072 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.500777 31699 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:03.483523 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.494041 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.500989 1142 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20250831 07:59:03.501406 1066 raft_consensus.cc:2747] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:03.503064 1084 raft_consensus.cc:2802] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:03.503551 1157 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.465327 1044 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.473590 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:03.519537 31801 raft_consensus.cc:1273] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:03.501040 1130 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:03.552371 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:03.553159 31809 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:03.554941 32435 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:03.557121 951 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.557626 951 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:03.558122 951 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.551050 1072 raft_consensus.cc:2747] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:03.562191 1013 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:03.564088 1087 log.cc:927] Time spent T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 1.917s user 0.004s sys 0.000s
I20250831 07:59:03.566040 1032 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.497408 954 consensus_queue.cc:1046] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20250831 07:59:03.523129 1100 raft_consensus.cc:695] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:03.574405 31727 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:03.496598 1085 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.580467 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 294) took 2568 ms. Trace:
I20250831 07:59:03.581856 1085 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.499532 1008 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:03.529453 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:03.504963 1138 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:03.532153 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.532778 1036 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:03.523692 31732 raft_consensus.cc:2466] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:59:03.505995 1061 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.583094 31730 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:03.531169 1084 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:03.517716 32464 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:03.576694 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.591181 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.595467 31733 raft_consensus.cc:2391] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:03.459316 31731 rpcz_store.cc:276] 0831 07:59:01.666933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.793732 (+126799us) service_pool.cc:225] Handling call
0831 07:59:01.851655 (+ 57923us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:01.910772 (+ 59117us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:01.913408 (+ 2636us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:02.965036 (+1051628us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x7b2c00027a08. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce69a09a1 00007fdce69affd2 00007fdce8039021 00007fdce7ffd3fa 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32
0831 07:59:02.965570 (+ 534us) log.cc:844] Serialized 26 byte log entry
0831 07:59:02.972047 (+ 6477us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:02.972126 (+ 79us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:02.972232 (+ 106us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:03.247593 (+275361us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:03.445107 (+197514us) raft_consensus.cc:1683] finished
0831 07:59:03.445132 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:03.445373 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":169604,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":23402,"spinlock_wait_cycles":79367808,"thread_start_us":5889,"threads_started":1,"wal-append.queue_time_us":1509}
I20250831 07:59:03.575588 909 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:03.598654 31727 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:03.540759 1157 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.602988 1110 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:03.605083 1161 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.573757 32465 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:03.606293 31729 raft_consensus.cc:2366] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5.
I20250831 07:59:03.602167 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.608593 31847 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:03.597177 1084 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:03.609670 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.615675 31733 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:03.617635 954 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:03.618187 954 raft_consensus.cc:695] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:03.599768 909 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:03.620342 909 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:03.623006 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:03.619074 954 consensus_queue.cc:237] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.634567 942 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.636103 1110 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:03.606263 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:03.594730 1036 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.611552 951 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:03.623308 907 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.625669 31731 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:03.655488 31726 raft_consensus.cc:1238] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 07:59:03.644605 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:03.660248 31727 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:03.635736 1138 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.659446 31733 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:03.645018 1013 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:03.669636 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1514) took 1424 ms. Trace:
I20250831 07:59:03.671296 31878 rpcz_store.cc:276] 0831 07:59:02.244931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:02.264822 (+ 19891us) service_pool.cc:225] Handling call
0831 07:59:02.419380 (+154558us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:02.739356 (+319976us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:02.739385 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:02.739428 (+ 43us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:02.739480 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:03.669335 (+929855us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b5400058054. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:03.669404 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:03.669564 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":724616064}
I20250831 07:59:03.674788 1036 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.654963 1157 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.660713 32463 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:03.709867 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:03.653995 1110 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:03.680163 995 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.000s sys 0.026s
I20250831 07:59:03.725927 31847 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:03.665050 1100 consensus_queue.cc:237] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.581451 32466 rpcz_store.cc:276] 0831 07:59:01.011949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.012136 (+ 187us) service_pool.cc:225] Handling call
0831 07:59:01.012476 (+ 340us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:01.026654 (+ 14178us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:01.026685 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:01.027164 (+ 479us) log.cc:844] Serialized 26 byte log entry
0831 07:59:01.217918 (+190754us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:01.245698 (+ 27780us) log.cc:844] Serialized 14 byte log entry
0831 07:59:01.309702 (+ 64004us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:01.309795 (+ 93us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:03.425595 (+2115800us) spinlock_profiling.cc:243] Waited 3.12 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:03.568051 (+142456us) raft_consensus.cc:1683] finished
0831 07:59:03.580203 (+ 12152us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:03.580384 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54492,"prepare.queue_time_us":13703,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":64057,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":26806,"spinlock_wait_cycles":7176576,"thread_start_us":178029,"threads_started":1,"wal-append.queue_time_us":454207}
I20250831 07:59:03.759917 31803 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
W20250831 07:59:03.725217 995 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s user 0.000s sys 0.026s
W20250831 07:59:03.730666 1136 consensus_meta.cc:287] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.500s user 0.011s sys 0.000s
I20250831 07:59:03.768101 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:03.795032 31878 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 4: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
W20250831 07:59:03.707235 1005 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.518s user 0.011s sys 0.021s
I20250831 07:59:03.764292 1110 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:03.764743 1044 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:03.777675 31727 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
W20250831 07:59:03.778752 31879 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.012s sys 0.018s
W20250831 07:59:03.820133 31879 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s user 0.013s sys 0.019s
W20250831 07:59:03.791030 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 65192.7us
I20250831 07:59:03.769091 942 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:03.805027 1011 consensus_meta.cc:287] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.548s user 0.007s sys 0.002s
W20250831 07:59:03.810549 1150 log.cc:927] Time spent T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.867s user 0.000s sys 0.003s
I20250831 07:59:03.812211 31730 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:03.773718 1161 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:03.794981 1013 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:03.790766 1061 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.826926 1138 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.827318 31731 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:03.833428 1013 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:03.833854 1013 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:03.838104 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:03.828644 1011 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:03.845201 1011 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.847577 32470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47886 (request call id 332) took 1798 ms. Trace:
I20250831 07:59:03.847976 32470 rpcz_store.cc:276] 0831 07:59:02.049618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:02.049738 (+ 120us) service_pool.cc:225] Handling call
0831 07:59:02.972539 (+922801us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:02.972783 (+ 244us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:03.120478 (+147695us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:03.847333 (+726855us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b540068b354. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:03.847495 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":341896320}
I20250831 07:59:03.848163 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:03.849082 31878 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:03.850792 31805 raft_consensus.cc:2391] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:03.818383 1136 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.864900 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:03.866946 31878 raft_consensus.cc:2391] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
W20250831 07:59:03.897907 31728 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.764s user 0.027s sys 0.016s
W20250831 07:59:03.898458 31728 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.765s user 0.027s sys 0.017s
I20250831 07:59:03.845536 1008 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:03.869709 939 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.025s sys 0.063s
I20250831 07:59:03.917505 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1550) took 1230 ms. Trace:
I20250831 07:59:03.918859 31806 rpcz_store.cc:276] 0831 07:59:02.687242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:02.689520 (+ 2278us) service_pool.cc:225] Handling call
0831 07:59:02.713422 (+ 23902us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:02.713800 (+ 378us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:02.713839 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:02.721943 (+ 8104us) log.cc:844] Serialized 26 byte log entry
0831 07:59:02.824658 (+102715us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:02.824707 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:02.824771 (+ 64us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:03.809880 (+985109us) spinlock_profiling.cc:243] Waited 54.6 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:03.904655 (+ 94775us) raft_consensus.cc:1683] finished
0831 07:59:03.904683 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:03.917429 (+ 12746us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2734,"raft.queue_time_us":3312,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":23185,"spinlock_wait_cycles":125656320,"thread_start_us":31441,"threads_started":1,"wal-append.queue_time_us":164879}
I20250831 07:59:03.924288 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:03.925465 31878 raft_consensus.cc:2408] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250831 07:59:03.924950 939 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s user 0.025s sys 0.064s
I20250831 07:59:03.864347 909 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.946153 31804 raft_consensus.cc:1238] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:03.923532 1013 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.956920 1136 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:03.958163 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1111) took 2594 ms. Trace:
I20250831 07:59:03.958706 31726 rpcz_store.cc:276] 0831 07:59:01.363523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.379632 (+ 16109us) service_pool.cc:225] Handling call
0831 07:59:03.162381 (+1782749us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b5800000f98. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:03.183384 (+ 21003us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:03.664853 (+481469us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:03.957757 (+292904us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b5400311d54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:03.958012 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1073440768}
I20250831 07:59:03.965509 32462 raft_consensus.cc:1238] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:03.969170 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:03.970012 31806 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:03.995570 31882 raft_consensus.cc:3053] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:59:03.996021 31882 raft_consensus.cc:738] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:03.996809 31882 consensus_queue.cc:260] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:03.996948 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:03.997951 31882 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:59:03.998126 32469 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:59:04.000033 31881 raft_consensus.cc:1273] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:04.001746 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1546) took 1759 ms. Trace:
I20250831 07:59:04.002121 31800 rpcz_store.cc:276] 0831 07:59:02.242526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:02.252095 (+ 9569us) service_pool.cc:225] Handling call
0831 07:59:02.420621 (+168526us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:02.490044 (+ 69423us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 07:59:02.771411 (+281367us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b1800297160. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce507e0f7 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce5098805 00007fdceb9e2327 00007fdce8007957 00007fdce8010595 00007fdce8010518
0831 07:59:02.998919 (+227508us) log.cc:844] Serialized 14 byte log entry
0831 07:59:03.138934 (+140015us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:03.140304 (+ 1370us) log.cc:844] Serialized 26 byte log entry
0831 07:59:03.140427 (+ 123us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:03.143937 (+ 3510us) log.cc:844] Serialized 14 byte log entry
0831 07:59:03.157098 (+ 13161us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:03.285755 (+128657us) spinlock_profiling.cc:243] Waited 8.07 ms on lock 0x7b1800297160. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b8bf 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:59:04.000913 (+715158us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b1800297160. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce5069786 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89
0831 07:59:04.001481 (+ 568us) spinlock_profiling.cc:243] Waited 68.8 ms on lock 0x7b54000852d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:04.001523 (+ 42us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:04.001566 (+ 43us) raft_consensus.cc:1683] finished
0831 07:59:04.001579 (+ 13us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:04.001694 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46879,"prepare.queue_time_us":104124,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":4101,"raft.queue_time_us":12590,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":27073,"spinlock_wait_cycles":198420352,"thread_start_us":366973,"threads_started":2,"wal-append.queue_time_us":370069}
I20250831 07:59:04.014029 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 300) took 1245 ms. Trace:
I20250831 07:59:04.014496 32467 rpcz_store.cc:276] 0831 07:59:02.768584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:02.768738 (+ 154us) service_pool.cc:225] Handling call
0831 07:59:02.769092 (+ 354us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:02.882804 (+113712us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:02.882849 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:02.882909 (+ 60us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:02.882951 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:03.995054 (+1112103us) spinlock_profiling.cc:243] Waited 4.41 ms on lock 0x7b1800324760. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce5069786 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89
0831 07:59:04.013778 (+ 18724us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b540066b2d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:04.013827 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:04.013968 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2571,"raft.queue_time_us":2709,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":17712,"spinlock_wait_cycles":264802048}
I20250831 07:59:03.987357 909 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:04.018898 1152 log.cc:927] Time spent T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 1.076s user 0.007s sys 0.000s
I20250831 07:59:04.044924 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1133) took 1235 ms. Trace:
I20250831 07:59:04.045493 31874 rpcz_store.cc:276] 0831 07:59:02.809408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:02.810001 (+ 593us) service_pool.cc:225] Handling call
0831 07:59:02.810234 (+ 233us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:02.817430 (+ 7196us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:02.817466 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:02.826482 (+ 9016us) log.cc:844] Serialized 26 byte log entry
0831 07:59:03.206550 (+380068us) spinlock_profiling.cc:243] Waited 5.78 ms on lock 0x7b18002b5e20. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce507e0f7 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdce7c3b2bb 00007fdce7c40dc3 00007fdce7c41171 00007fdce7fd009f
0831 07:59:03.459637 (+253087us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:03.459678 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:03.459769 (+ 91us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:04.044555 (+584786us) raft_consensus.cc:1683] finished
0831 07:59:04.044586 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:04.044847 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6844,"raft.queue_time_us":17083,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":31056,"spinlock_wait_cycles":13288704,"thread_start_us":252879,"threads_started":1,"wal-append.queue_time_us":1214495}
W20250831 07:59:04.367343 31848 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.075s user 0.004s sys 0.062s
W20250831 07:59:04.396226 31848 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.104s user 0.005s sys 0.062s
I20250831 07:59:04.397260 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1121) took 1390 ms. Trace:
I20250831 07:59:04.397735 31725 rpcz_store.cc:276] 0831 07:59:03.006661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.006830 (+ 169us) service_pool.cc:225] Handling call
0831 07:59:04.397085 (+1390255us) spinlock_profiling.cc:243] Waited 85.3 ms on lock 0x7b5400090454. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:04.397197 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":196144000}
I20250831 07:59:04.392436 31882 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
W20250831 07:59:04.402483 31882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1130) took 1594 ms (client timeout 1756 ms). Trace:
W20250831 07:59:04.402964 31882 rpcz_store.cc:269] 0831 07:59:02.808434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:02.808553 (+ 119us) service_pool.cc:225] Handling call
0831 07:59:03.995426 (+1186873us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b5400199720. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7f8c550 00007fdce8000dc8 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106
0831 07:59:04.402411 (+406985us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":777837952}
I20250831 07:59:04.407656 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:04.412118 31874 raft_consensus.cc:1273] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:59:04.412374 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1572) took 1179 ms. Trace:
I20250831 07:59:04.412856 31729 rpcz_store.cc:276] 0831 07:59:03.232924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.458303 (+225379us) service_pool.cc:225] Handling call
0831 07:59:04.411079 (+952776us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b5400010c54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:04.412298 (+ 1219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":302470912}
I20250831 07:59:04.409184 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:04.414355 31881 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 5
I20250831 07:59:04.415792 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1440) took 1055 ms. Trace:
W20250831 07:59:04.416083 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 279670us
I20250831 07:59:04.416206 31805 rpcz_store.cc:276] 0831 07:59:03.355402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.355543 (+ 141us) service_pool.cc:225] Handling call
0831 07:59:04.410465 (+1054922us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b54000cf3d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:04.410620 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1105164672}
W20250831 07:59:04.064615 992 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.807s user 0.015s sys 0.013s
I20250831 07:59:04.421314 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:04.422035 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1450) took 1009 ms. Trace:
I20250831 07:59:04.421330 31880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:50826 (request call id 12) took 1593 ms. Trace:
I20250831 07:59:04.422482 31804 rpcz_store.cc:276] 0831 07:59:03.412597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.412720 (+ 123us) service_pool.cc:225] Handling call
0831 07:59:03.413134 (+ 414us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:03.947210 (+534076us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:04.421830 (+474620us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b54001d9f54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:04.421981 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1225184128}
I20250831 07:59:04.424846 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:04.426343 31804 raft_consensus.cc:2391] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
W20250831 07:59:04.428108 1118 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20250831 07:59:04.428639 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:04.035640 31806 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
W20250831 07:59:04.519207 31875 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.226s user 0.005s sys 0.050s
W20250831 07:59:04.519913 31875 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.227s user 0.005s sys 0.050s
I20250831 07:59:04.517215 31800 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:04.521728 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:04.423202 31880 rpcz_store.cc:276] 0831 07:59:02.821892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:02.875750 (+ 53858us) service_pool.cc:225] Handling call
0831 07:59:04.085540 (+1209790us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b54003ef154. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ff7813 00007fdcecf5d1ba 00007fdce65f71c4 00007fdce65f7152 00007fdce65f7081 00007fdce65f6ffc 00007fdce65f62b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:04.415690 (+330150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":928375680}
W20250831 07:59:04.420090 992 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.163s user 0.016s sys 0.014s
W20250831 07:59:04.446554 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:04.404405 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:04.443495 1084 consensus_meta.cc:287] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.821s user 0.013s sys 0.002s
I20250831 07:59:04.473791 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:04.536038 31805 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
W20250831 07:59:04.483192 31808 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.329s user 0.016s sys 0.035s
W20250831 07:59:04.483887 31883 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.187s user 0.016s sys 0.010s
W20250831 07:59:04.538000 31883 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.245s user 0.017s sys 0.010s
W20250831 07:59:04.491083 31849 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.200s user 0.007s sys 0.024s
I20250831 07:59:04.468794 974 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:04.510097 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:04.483681 1092 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.147s user 0.012s sys 0.039s
I20250831 07:59:04.515578 1008 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:04.523926 31806 raft_consensus.cc:2408] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:04.546934 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:04.548031 31876 raft_consensus.cc:2408] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:04.548520 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:04.549279 31880 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:59:04.552197 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:04.533577 31881 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
W20250831 07:59:04.539673 31849 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.248s user 0.007s sys 0.024s
W20250831 07:59:04.540099 31808 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.409s user 0.017s sys 0.035s
I20250831 07:59:04.555943 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:04.541853 32435 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
W20250831 07:59:04.516222 1018 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.227s user 0.019s sys 0.041s
W20250831 07:59:04.642782 1018 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.354s user 0.020s sys 0.041s
W20250831 07:59:04.649755 1153 log.cc:927] Time spent T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 1.708s user 0.000s sys 0.010s
I20250831 07:59:04.553769 992 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:59:04.553483 1092 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.212s user 0.012s sys 0.039s
I20250831 07:59:04.552933 31800 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:04.557713 31882 raft_consensus.cc:2391] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:04.554071 31848 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:04.557518 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:04.519142 31774 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.058s user 0.014s sys 0.035s
W20250831 07:59:04.676244 31774 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.222s user 0.014s sys 0.035s
I20250831 07:59:04.525858 1112 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:04.558521 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:04.587782 909 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:04.588079 32462 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47886 (request call id 344) took 1217 ms. Trace:
I20250831 07:59:04.544512 1110 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:04.590139 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:04.698290 31881 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:04.698690 31880 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:04.699188 31880 raft_consensus.cc:604] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:04.699505 31880 raft_consensus.cc:612] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:59:04.544124 974 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:04.595688 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:04.700456 31874 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:04.597046 31849 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:04.598855 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:04.701176 31628 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59666 (request call id 235) took 7179 ms. Trace:
I20250831 07:59:04.701864 31876 raft_consensus.cc:2408] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:04.629158 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:04.543977 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 133) took 1245 ms. Trace:
W20250831 07:59:04.704723 31773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.243s user 0.028s sys 0.042s
W20250831 07:59:04.705169 31773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.244s user 0.028s sys 0.042s
I20250831 07:59:04.554234 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:04.583058 1002 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:04.680469 31877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1131) took 1871 ms. Trace:
W20250831 07:59:04.681775 1175 log.cc:927] Time spent T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250831 07:59:04.553335 1013 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:04.624398 1039 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:04.685537 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1125) took 1578 ms. Trace:
I20250831 07:59:04.712024 31728 rpcz_store.cc:276] 0831 07:59:03.106872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.107018 (+ 146us) service_pool.cc:225] Handling call
0831 07:59:03.107354 (+ 336us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:03.107670 (+ 316us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:03.107699 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:03.114495 (+ 6796us) log.cc:844] Serialized 26 byte log entry
0831 07:59:03.899007 (+784512us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:03.899063 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:03.943726 (+ 44663us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:04.685269 (+741543us) raft_consensus.cc:1683] finished
0831 07:59:04.685298 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:04.685469 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":201,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":7207,"thread_start_us":765100,"threads_started":1,"wal-append.queue_time_us":1548565}
W20250831 07:59:04.686383 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 132397us
I20250831 07:59:04.714643 31848 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:04.628988 1044 consensus_queue.cc:1035] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20250831 07:59:04.701277 909 raft_consensus.cc:695] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:04.696913 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:04.702975 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 106039us
I20250831 07:59:04.650913 992 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:04.691992 1018 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:04.706014 31804 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:04.707504 31775 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.246s user 0.018s sys 0.039s
I20250831 07:59:04.698063 995 raft_consensus.cc:2802] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:04.705899 31877 rpcz_store.cc:276] 0831 07:59:02.808760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:02.809529 (+ 769us) service_pool.cc:225] Handling call
0831 07:59:02.809817 (+ 288us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:02.816007 (+ 6190us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:02.816037 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:02.816913 (+ 876us) log.cc:844] Serialized 26 byte log entry
0831 07:59:02.888759 (+ 71846us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:02.888798 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:02.888858 (+ 60us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:04.514409 (+1625551us) spinlock_profiling.cc:243] Waited 43.3 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:04.650671 (+136262us) raft_consensus.cc:1683] finished
0831 07:59:04.650703 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:04.680391 (+ 29688us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17397,"raft.queue_time_us":9278,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":1837,"spinlock_wait_cycles":99529600,"thread_start_us":71482,"threads_started":1,"wal-append.queue_time_us":124730}
I20250831 07:59:04.546190 1084 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:04.717964 31848 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:04.721055 31849 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:04.703306 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:04.632508 31627 catalog_manager.cc:5582] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 2 to 4, leader changed from 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 4 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:59:04.704792 1147 raft_consensus.cc:2747] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 07:59:04.723425 31775 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.262s user 0.018s sys 0.039s
W20250831 07:59:04.719238 1116 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.191s user 0.010s sys 0.010s
W20250831 07:59:04.717866 32471 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.196s user 0.006s sys 0.015s
I20250831 07:59:04.696909 1112 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:04.728227 31800 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
W20250831 07:59:04.722201 1174 log.cc:927] Time spent T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.243s user 0.001s sys 0.001s
I20250831 07:59:04.731320 31848 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:04.734426 32471 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.213s user 0.006s sys 0.015s
W20250831 07:59:04.709965 1067 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.234s user 0.048s sys 0.107s
W20250831 07:59:04.718179 914 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.202s user 0.008s sys 0.011s
W20250831 07:59:04.730718 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 1.26969e+06us
I20250831 07:59:04.721765 1118 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:04.701817 31628 rpcz_store.cc:276] 0831 07:58:57.518861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:57.518996 (+ 135us) service_pool.cc:225] Handling call
0831 07:59:04.698548 (+7179552us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:59:01.874164 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:59:02.123167 (+249003us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:59:02.126071 (+ 2904us) write_op.cc:435] Acquired schema lock
0831 07:59:02.131990 (+ 5919us) tablet.cc:660] Decoding operations
0831 07:59:02.240445 (+108455us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:59:02.252079 (+ 11634us) write_op.cc:641] Partition lock acquired for write op
0831 07:59:02.258582 (+ 6503us) tablet.cc:683] Acquiring locks for 3 operations
0831 07:59:02.433238 (+174656us) tablet.cc:699] Row locks acquired
0831 07:59:02.440117 (+ 6879us) write_op.cc:260] PREPARE: finished
0831 07:59:02.487145 (+ 47028us) write_op.cc:270] Start()
0831 07:59:02.516502 (+ 29357us) write_op.cc:276] Timestamp: P: 1756627142474642 usec, L: 0
0831 07:59:02.522478 (+ 5976us) op_driver.cc:348] REPLICATION: starting
0831 07:59:02.595307 (+ 72829us) log.cc:844] Serialized 1308 byte log entry
0831 07:59:03.081719 (+486412us) op_driver.cc:464] REPLICATION: finished
0831 07:59:03.715516 (+633797us) write_op.cc:301] APPLY: starting
0831 07:59:03.715651 (+ 135us) tablet.cc:1365] starting BulkCheckPresence
0831 07:59:03.715852 (+ 201us) tablet.cc:1368] finished BulkCheckPresence
0831 07:59:03.719658 (+ 3806us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:59:03.720311 (+ 653us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:59:03.720544 (+ 233us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0831 07:59:03.720615 (+ 71us) write_op.cc:312] APPLY: finished
0831 07:59:03.724251 (+ 3636us) log.cc:844] Serialized 35 byte log entry
0831 07:59:03.724478 (+ 227us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:03.724576 (+ 98us) write_op.cc:454] Released schema lock
0831 07:59:03.724615 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0831 07:59:03.725005 (+ 390us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:03.725031 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":258586,"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":597052,"apply.run_cpu_time_us":2914,"apply.run_wall_time_us":15362,"num_ops":3,"prepare.queue_time_us":769754,"prepare.run_cpu_time_us":42113,"prepare.run_wall_time_us":843481,"raft.queue_time_us":275990,"raft.run_cpu_time_us":17030,"raft.run_wall_time_us":278755,"replication_time_us":559066,"thread_start_us":1146414,"threads_started":4,"wal-append.queue_time_us":161038,"wal-append.run_cpu_time_us":22920,"wal-append.run_wall_time_us":1950463}]]}
I20250831 07:59:04.696630 32462 rpcz_store.cc:276] 0831 07:59:03.370554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.618499 (+247945us) service_pool.cc:225] Handling call
0831 07:59:03.618851 (+ 352us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:03.966203 (+347352us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:04.587686 (+621483us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b54003ba954. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:04.587987 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":796578048}
I20250831 07:59:04.746078 31848 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:04.726467 1099 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.238s user 0.015s sys 0.078s
W20250831 07:59:04.733995 1116 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.207s user 0.010s sys 0.010s
I20250831 07:59:04.744621 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1139) took 1859 ms. Trace:
I20250831 07:59:04.776530 31879 rpcz_store.cc:276] 0831 07:59:02.885365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.232635 (+347270us) service_pool.cc:225] Handling call
0831 07:59:03.232963 (+ 328us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:03.233396 (+ 433us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:03.233425 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:03.233916 (+ 491us) log.cc:844] Serialized 26 byte log entry
0831 07:59:03.820549 (+586633us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:03.820591 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:03.820655 (+ 64us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:04.744415 (+923760us) raft_consensus.cc:1683] finished
0831 07:59:04.744437 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:04.744565 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":120,"thread_start_us":584074,"threads_started":1,"wal-append.queue_time_us":1283771}
I20250831 07:59:04.730391 995 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:04.749151 909 consensus_queue.cc:237] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:04.728845 31627 catalog_manager.cc:5582] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:59:04.704638 31731 rpcz_store.cc:276] 0831 07:59:03.295596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.674793 (+379197us) service_pool.cc:225] Handling call
0831 07:59:03.826964 (+152171us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:03.833652 (+ 6688us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:04.516470 (+682818us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b5400119ed0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:04.540854 (+ 24384us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":348348800}
I20250831 07:59:04.719281 1039 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:04.757539 1084 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:04.762789 1112 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:04.763981 1013 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:04.700971 1110 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:04.740938 1067 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.269s user 0.048s sys 0.108s
I20250831 07:59:04.717569 1002 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:04.775358 31800 raft_consensus.cc:1238] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:04.785038 995 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:04.787256 1044 raft_consensus.cc:2747] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:04.790426 1013 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:04.800004 978 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.192s user 0.015s sys 0.016s
I20250831 07:59:04.795557 1067 raft_consensus.cc:695] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:04.746996 1099 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.278s user 0.015s sys 0.079s
I20250831 07:59:04.743535 1118 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:04.732147 1018 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:04.790892 31698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.305s user 0.003s sys 0.022s
W20250831 07:59:04.818161 31698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.332s user 0.004s sys 0.023s
W20250831 07:59:04.789665 31699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.258s user 0.007s sys 0.015s
I20250831 07:59:04.788839 919 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:04.741242 914 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.236s user 0.009s sys 0.012s
I20250831 07:59:04.789077 1011 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:04.819365 31732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.109s user 0.012s sys 0.009s
W20250831 07:59:04.807063 1007 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.293s user 0.023s sys 0.023s
W20250831 07:59:04.803730 938 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.321s user 0.009s sys 0.014s
W20250831 07:59:04.806736 978 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.201s user 0.016s sys 0.017s
W20250831 07:59:04.830269 31732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.120s user 0.012s sys 0.010s
I20250831 07:59:04.832226 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:04.881165 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:04.818293 1118 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:04.836683 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:04.886023 31876 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:04.812309 1008 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:04.801852 1013 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:04.829684 31734 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.236s user 0.025s sys 0.047s
W20250831 07:59:04.892148 31734 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.299s user 0.025s sys 0.047s
I20250831 07:59:04.892027 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:04.802577 1147 consensus_queue.cc:1046] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20250831 07:59:04.780112 31627 catalog_manager.cc:5582] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 2, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 2 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:59:04.897522 31627 catalog_manager.cc:5582] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 2 to 3, leader changed from 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 3 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:59:04.812358 1060 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:04.881633 31879 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:04.818956 31699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.288s user 0.007s sys 0.015s
I20250831 07:59:04.806784 1181 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250831 07:59:04.883113 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 69839.6us
I20250831 07:59:04.883908 31880 raft_consensus.cc:1273] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:04.839599 919 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:04.842463 1011 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:04.898872 31627 catalog_manager.cc:5582] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250831 07:59:04.960937 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:04.976684 31881 raft_consensus.cc:1273] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:04.976467 31627 catalog_manager.cc:5582] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:04.841122 1180 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 07:59:04.977882 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 1.4771e+06us
I20250831 07:59:04.978631 31699 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:04.979482 31773 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
W20250831 07:59:04.880331 31802 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.298s user 0.010s sys 0.018s
W20250831 07:59:04.839427 1007 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.325s user 0.023s sys 0.024s
I20250831 07:59:04.889562 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:04.980655 31874 raft_consensus.cc:2391] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
W20250831 07:59:04.981271 31802 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.399s user 0.011s sys 0.018s
W20250831 07:59:04.881685 31801 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.269s user 0.012s sys 0.036s
I20250831 07:59:04.889555 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:04.982699 31877 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:59:04.842229 938 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.360s user 0.013s sys 0.014s
I20250831 07:59:04.892622 31882 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:59:04.983729 31801 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.371s user 0.012s sys 0.036s
I20250831 07:59:04.809201 31804 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:04.986080 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:04.990366 31879 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
W20250831 07:59:04.911113 1184 log.cc:927] Time spent T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20250831 07:59:04.914126 974 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:04.981212 1180 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:05.001706 1018 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:04.985608 1060 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:05.005057 1060 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:04.999286 31876 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:05.008023 1039 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:04.992736 978 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:05.009135 978 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:05.002986 974 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:05.001700 1180 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:05.016136 1116 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:05.016710 1116 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:05.017112 1116 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:05.011484 978 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:04.896303 992 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:04.900668 898 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 07:59:04.893384 954 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.091s user 0.010s sys 0.023s
I20250831 07:59:04.977335 1181 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:05.051342 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 71917.8us
I20250831 07:59:05.052577 31882 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
W20250831 07:59:05.054032 31809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.202s user 0.018s sys 0.024s
I20250831 07:59:04.896766 1008 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:05.067585 31877 raft_consensus.cc:2408] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:04.968163 1067 consensus_queue.cc:237] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:05.075381 31803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.234s user 0.012s sys 0.023s
W20250831 07:59:05.095505 31803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.254s user 0.012s sys 0.023s
W20250831 07:59:05.079643 31809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.227s user 0.018s sys 0.024s
W20250831 07:59:05.083396 31733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.245s user 0.002s sys 0.034s
W20250831 07:59:05.067878 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 72562.8us
I20250831 07:59:05.099171 31699 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:05.115350 31699 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:05.068332 1007 raft_consensus.cc:2802] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:05.104336 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:05.104696 1011 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
W20250831 07:59:05.075325 31700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.541s user 0.012s sys 0.029s
W20250831 07:59:05.123051 31700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.589s user 0.012s sys 0.029s
W20250831 07:59:05.075342 954 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.271s user 0.010s sys 0.023s
W20250831 07:59:05.073962 940 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.627s user 0.031s sys 0.131s
W20250831 07:59:05.097221 31733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.259s user 0.002s sys 0.034s
I20250831 07:59:05.087997 1110 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:05.075476 1181 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:05.118243 914 raft_consensus.cc:2747] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:05.122327 1084 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:05.118292 1007 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:05.080957 1002 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:05.131408 940 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.682s user 0.031s sys 0.131s
I20250831 07:59:05.087917 1180 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:05.143182 1180 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:05.075286 1118 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:05.123803 898 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:05.140992 919 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:05.138590 1007 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:05.137270 1084 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:05.149540 1185 log.cc:927] Time spent T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.177s user 0.000s sys 0.001s
I20250831 07:59:05.147306 898 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:05.149914 31726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.147s user 0.014s sys 0.026s
I20250831 07:59:05.146647 978 consensus_queue.cc:1035] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.022s
I20250831 07:59:05.157508 1084 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:05.117184 31699 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:05.223810 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 106731us
W20250831 07:59:05.219187 31726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.217s user 0.014s sys 0.026s
W20250831 07:59:05.247967 1142 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.412s user 0.011s sys 0.045s
W20250831 07:59:05.248394 1142 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.413s user 0.011s sys 0.045s
W20250831 07:59:05.256093 31773 leader_election.cc:336] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.643s (SENT)
I20250831 07:59:05.256824 31773 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:05.257849 940 raft_consensus.cc:2747] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 07:59:05.260043 32436 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.867s user 0.006s sys 0.027s
W20250831 07:59:05.260449 32436 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.867s user 0.006s sys 0.027s
W20250831 07:59:05.263809 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 871028us
I20250831 07:59:05.245476 1008 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:05.274895 31699 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:05.241549 1188 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:05.280494 1180 consensus_queue.cc:1046] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20250831 07:59:05.290256 938 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:05.294509 1060 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:05.320824 1100 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.513s user 0.030s sys 0.026s
W20250831 07:59:05.372849 1100 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.565s user 0.030s sys 0.026s
I20250831 07:59:05.372735 1060 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:05.373704 31807 raft_consensus.cc:1238] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:59:05.375591 1060 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:05.326478 1189 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:05.334988 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1116) took 3402 ms. Trace:
I20250831 07:59:05.345706 1008 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:05.346450 1194 log.cc:927] Time spent T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20250831 07:59:05.346987 1193 log.cc:927] Time spent T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20250831 07:59:05.349416 1002 raft_consensus.cc:3053] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:05.355187 1192 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:05.297900 914 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:05.295775 978 consensus_queue.cc:1035] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20250831 07:59:05.367954 940 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:05.369834 1013 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:05.371393 1181 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:05.282482 1188 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:05.387576 31734 rpcz_store.cc:276] 0831 07:59:01.932882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:02.561364 (+628482us) service_pool.cc:225] Handling call
0831 07:59:02.561695 (+ 331us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:02.918345 (+356650us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:02.918402 (+ 57us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:03.587887 (+669485us) spinlock_profiling.cc:243] Waited 22.8 ms on lock 0x7b2c00027a08. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce69a09a1 00007fdce69affd2 00007fdce8039021 00007fdce7ffd3fa 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32
0831 07:59:03.588231 (+ 344us) log.cc:844] Serialized 26 byte log entry
0831 07:59:04.892537 (+1304306us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:04.892574 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:04.892641 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:05.334624 (+441983us) raft_consensus.cc:1683] finished
0831 07:59:05.334655 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:05.334909 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":356201,"raft.queue_time_us":357850,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":166,"spinlock_wait_cycles":52418432,"thread_start_us":1299211,"threads_started":1,"wal-append.queue_time_us":1738229}
I20250831 07:59:05.384202 938 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:05.342638 1118 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:05.343845 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:05.390340 1200 log.cc:927] Time spent T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20250831 07:59:05.319037 31730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.462s user 0.026s sys 0.115s
W20250831 07:59:05.400686 31730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.544s user 0.026s sys 0.115s
I20250831 07:59:05.401566 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1132) took 2048 ms. Trace:
I20250831 07:59:05.392232 1002 raft_consensus.cc:738] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:05.402048 31730 rpcz_store.cc:276] 0831 07:59:03.353615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.828425 (+474810us) service_pool.cc:225] Handling call
0831 07:59:03.837417 (+ 8992us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:03.837605 (+ 188us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 07:59:03.856906 (+ 19301us) log.cc:844] Serialized 14 byte log entry
0831 07:59:05.401195 (+1544289us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:05.401257 (+ 62us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:05.401312 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:05.401387 (+ 75us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:05.401511 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":31970,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"thread_start_us":1543956,"threads_started":1,"wal-append.queue_time_us":1461939}
I20250831 07:59:05.395246 914 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:05.395812 940 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:05.393190 1189 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:05.440243 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1578) took 2205 ms. Trace:
I20250831 07:59:05.440833 31732 rpcz_store.cc:276] 0831 07:59:03.234809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.625873 (+391064us) service_pool.cc:225] Handling call
0831 07:59:03.646563 (+ 20690us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:03.646918 (+ 355us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:03.646946 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:03.647408 (+ 462us) log.cc:844] Serialized 26 byte log entry
0831 07:59:04.882378 (+1234970us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:04.882434 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:04.882518 (+ 84us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:05.440007 (+557489us) raft_consensus.cc:1683] finished
0831 07:59:05.440032 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:05.440182 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":649,"raft.queue_time_us":83,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":2770,"spinlock_wait_cycles":27520,"thread_start_us":1171676,"threads_started":1,"wal-append.queue_time_us":1725039}
I20250831 07:59:05.442427 1189 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:05.443217 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:05.441883 940 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:05.444797 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 164) took 1601 ms. Trace:
I20250831 07:59:05.389984 1008 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:05.452894 31727 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.584s user 0.018s sys 0.045s
I20250831 07:59:05.448834 31809 rpcz_store.cc:276] 0831 07:59:03.843516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.843659 (+ 143us) service_pool.cc:225] Handling call
0831 07:59:03.844037 (+ 378us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:03.844287 (+ 250us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:03.844323 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:03.852090 (+ 7767us) log.cc:844] Serialized 26 byte log entry
0831 07:59:05.096454 (+1244364us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:05.096511 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:05.096604 (+ 93us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:05.444536 (+347932us) raft_consensus.cc:1683] finished
0831 07:59:05.444564 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:05.444727 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1244007,"threads_started":1,"wal-append.queue_time_us":1586015}
W20250831 07:59:05.453389 31727 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.584s user 0.019s sys 0.045s
I20250831 07:59:05.363323 1116 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:05.452543 31875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.803s user 0.010s sys 0.037s
W20250831 07:59:05.458057 31875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s user 0.010s sys 0.038s
W20250831 07:59:05.465847 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:05.408918 1192 raft_consensus.cc:2747] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 07:59:05.486013 31878 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.463s user 0.014s sys 0.034s
W20250831 07:59:05.487926 31878 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.466s user 0.014s sys 0.034s
W20250831 07:59:05.480160 31808 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.850s user 0.015s sys 0.031s
I20250831 07:59:05.476742 32466 raft_consensus.cc:2391] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
W20250831 07:59:05.489075 31775 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.008s sys 0.004s
W20250831 07:59:05.489488 31808 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.860s user 0.016s sys 0.031s
W20250831 07:59:05.477443 995 consensus_meta.cc:287] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.673s user 0.003s sys 0.009s
W20250831 07:59:05.499162 31775 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.527s user 0.008s sys 0.004s
W20250831 07:59:05.519229 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 550567us
W20250831 07:59:05.486213 31883 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.856s user 0.018s sys 0.028s
W20250831 07:59:05.520267 31883 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.891s user 0.018s sys 0.029s
I20250831 07:59:05.524950 31808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1555) took 2632 ms. Trace:
I20250831 07:59:05.525539 31808 rpcz_store.cc:276] 0831 07:59:02.866786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:02.868039 (+ 1253us) service_pool.cc:225] Handling call
0831 07:59:02.888756 (+ 20717us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:03.120515 (+231759us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:03.120555 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:03.120968 (+ 413us) log.cc:844] Serialized 26 byte log entry
0831 07:59:04.555498 (+1434530us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:05.498528 (+943030us) log.cc:844] Serialized 14 byte log entry
0831 07:59:05.498820 (+ 292us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:05.498907 (+ 87us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:05.498944 (+ 37us) raft_consensus.cc:1683] finished
0831 07:59:05.498956 (+ 12us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:05.499080 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55864,"raft.queue_time_us":104387,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":25031,"thread_start_us":2524485,"threads_started":3,"wal-append.queue_time_us":1685334}
I20250831 07:59:05.536721 1018 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:05.537611 1018 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:05.543026 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:05.543979 31775 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:05.545400 1099 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:05.545938 1099 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:05.546418 1099 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:05.552366 31775 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:05.555562 1191 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:05.556049 1191 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:05.541695 1018 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:05.556414 1191 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:05.518244 1123 raft_consensus.cc:2747] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:05.499431 978 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:05.566562 31849 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.837s user 0.015s sys 0.020s
I20250831 07:59:05.520853 993 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:05.524292 1189 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:59:05.569720 31849 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.840s user 0.016s sys 0.020s
W20250831 07:59:05.577891 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 848167us
W20250831 07:59:05.574113 32471 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.831s user 0.014s sys 0.040s
W20250831 07:59:05.585067 32471 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.842s user 0.014s sys 0.040s
W20250831 07:59:05.569707 1199 log.cc:927] Time spent T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.124s user 0.000s sys 0.001s
I20250831 07:59:05.532385 995 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:05.556035 1190 consensus_queue.cc:1046] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20250831 07:59:05.570261 1187 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:05.488569 1002 consensus_queue.cc:260] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:05.596093 31849 connection.cc:232] RPC call timeout handler was delayed by -1.3797047560000237s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17410000000000003s for a response
I20250831 07:59:05.663004 32471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38366 (request call id 393) took 2260 ms. Trace:
W20250831 07:59:05.669394 31849 connection.cc:232] RPC call timeout handler was delayed by -1.3289231010001004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22290000000000001s for a response
W20250831 07:59:05.669919 31849 connection.cc:232] RPC call timeout handler was delayed by -1.06290475600008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2093s for a response
W20250831 07:59:05.597160 909 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.000s sys 0.021s
I20250831 07:59:05.590997 1118 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:05.594729 31848 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.019s sys 0.059s
I20250831 07:59:05.584879 1189 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:05.674356 1180 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:05.675282 32466 raft_consensus.cc:1273] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250831 07:59:05.676223 1188 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:05.677222 898 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:05.596679 954 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:05.595767 1142 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:05.679977 1181 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:05.680619 1187 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:05.681632 1002 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:05.615007 1110 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:05.582291 993 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:05.683593 1201 log.cc:927] Time spent T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.278s user 0.000s sys 0.000s
W20250831 07:59:05.684295 1209 log.cc:927] Time spent T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20250831 07:59:05.684621 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1448) took 2285 ms. Trace:
I20250831 07:59:05.669479 32471 rpcz_store.cc:276] 0831 07:59:03.402861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.403060 (+ 199us) service_pool.cc:225] Handling call
0831 07:59:03.442142 (+ 39082us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:03.521172 (+ 79030us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:03.521199 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:03.521554 (+ 355us) log.cc:844] Serialized 26 byte log entry
0831 07:59:04.740648 (+1219094us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:05.585699 (+845051us) log.cc:844] Serialized 14 byte log entry
0831 07:59:05.588106 (+ 2407us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:05.597362 (+ 9256us) spinlock_profiling.cc:243] Waited 78.7 ms on lock 0x7b540066bf54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:05.597433 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:05.597514 (+ 81us) raft_consensus.cc:1683] finished
0831 07:59:05.662801 (+ 65287us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:05.662947 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11206,"prepare.queue_time_us":156217,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":2225,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":3999,"spinlock_wait_cycles":180923776,"thread_start_us":2061344,"threads_started":2,"wal-append.queue_time_us":1652385}
W20250831 07:59:05.660081 1092 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.957s user 0.000s sys 0.044s
I20250831 07:59:05.581730 978 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:05.682386 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1133) took 2328 ms. Trace:
W20250831 07:59:05.711364 31733 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.014s sys 0.009s
W20250831 07:59:05.929590 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:06.000288 1099 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:06.003535 1207 log.cc:927] Time spent T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.344s user 0.001s sys 0.000s
W20250831 07:59:05.933950 31773 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 2.365s (SENT) [suppressed 19 similar messages]
I20250831 07:59:05.992259 1112 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:05.932550 31848 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.186s user 0.019s sys 0.079s
I20250831 07:59:06.004841 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 161) took 2209 ms. Trace:
W20250831 07:59:06.005164 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 1.25907e+06us
I20250831 07:59:06.004383 1018 raft_consensus.cc:3053] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:05.831459 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1531) took 2427 ms. Trace:
W20250831 07:59:05.667387 1032 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.010s sys 0.012s
W20250831 07:59:06.004443 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 70531.3us
I20250831 07:59:06.006356 31848 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:05.740576 31700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.000s sys 0.025s
W20250831 07:59:05.693378 31627 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.687s user 0.006s sys 0.021s
W20250831 07:59:05.869009 31698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.738s user 0.011s sys 0.019s
I20250831 07:59:06.005285 31803 rpcz_store.cc:276] 0831 07:59:03.795877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.813109 (+ 17232us) service_pool.cc:225] Handling call
0831 07:59:03.813551 (+ 442us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:03.813807 (+ 256us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:03.813843 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:03.820534 (+ 6691us) log.cc:844] Serialized 26 byte log entry
0831 07:59:05.097859 (+1277325us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:05.097915 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:05.098025 (+ 110us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:06.002969 (+904944us) spinlock_profiling.cc:243] Waited 2.76 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:06.004526 (+ 1557us) raft_consensus.cc:1683] finished
0831 07:59:06.004545 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:06.004771 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6343936,"thread_start_us":1256381,"threads_started":1,"wal-append.queue_time_us":1728534}
W20250831 07:59:05.884595 32467 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.004s sys 0.020s
W20250831 07:59:05.932708 31733 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.803s user 0.016s sys 0.014s
I20250831 07:59:05.907933 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1528) took 2793 ms. Trace:
W20250831 07:59:06.020527 31698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.889s user 0.011s sys 0.028s
W20250831 07:59:06.009505 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 254483us
I20250831 07:59:05.947546 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1131) took 2594 ms. Trace:
I20250831 07:59:05.684981 1013 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:06.020787 32467 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s user 0.011s sys 0.021s
W20250831 07:59:06.020282 31627 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.014s user 0.007s sys 0.037s
W20250831 07:59:06.024818 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 893540us
W20250831 07:59:05.817718 1044 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.784s user 0.010s sys 0.025s
W20250831 07:59:05.903579 31699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.010s sys 0.012s
W20250831 07:59:06.049027 31699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.771s user 0.011s sys 0.017s
W20250831 07:59:06.049546 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 774730us
W20250831 07:59:06.020135 31700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.895s user 0.000s sys 0.035s
W20250831 07:59:05.908313 1067 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.068s sys 0.129s
I20250831 07:59:06.020807 31802 rpcz_store.cc:276] 0831 07:59:03.399254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.399399 (+ 145us) service_pool.cc:225] Handling call
0831 07:59:03.399771 (+ 372us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:03.521029 (+121258us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:03.521060 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:03.551856 (+ 30796us) log.cc:844] Serialized 26 byte log entry
0831 07:59:04.981722 (+1429866us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:04.981760 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:05.086719 (+104959us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400028d54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:05.086791 (+ 72us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:05.684319 (+597528us) raft_consensus.cc:1683] finished
0831 07:59:05.684351 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:05.684563 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":78718,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":14014,"spinlock_wait_cycles":265452928,"thread_start_us":1398922,"threads_started":1,"wal-append.queue_time_us":1807383}
I20250831 07:59:06.024515 31883 rpcz_store.cc:276] 0831 07:59:03.114166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.269657 (+155491us) service_pool.cc:225] Handling call
0831 07:59:03.272421 (+ 2764us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:04.538721 (+1266300us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:04.538765 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:04.539344 (+ 579us) log.cc:844] Serialized 26 byte log entry
0831 07:59:05.520636 (+981292us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:05.520680 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:05.520760 (+ 80us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:05.907582 (+386822us) raft_consensus.cc:1683] finished
0831 07:59:05.907612 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:05.907806 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20057,"raft.queue_time_us":38763,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":115,"thread_start_us":2136719,"threads_started":2,"wal-append.queue_time_us":1301002}
W20250831 07:59:06.059569 1067 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s user 0.079s sys 0.143s
I20250831 07:59:05.747174 1180 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:06.060883 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 157) took 2491 ms. Trace:
I20250831 07:59:06.008216 1018 raft_consensus.cc:738] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:06.061275 31801 rpcz_store.cc:276] 0831 07:59:03.569285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.569555 (+ 270us) service_pool.cc:225] Handling call
0831 07:59:03.585724 (+ 16169us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:03.586121 (+ 397us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:03.586162 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:03.586700 (+ 538us) log.cc:844] Serialized 26 byte log entry
0831 07:59:04.984164 (+1397464us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:04.984202 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:04.984271 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:06.060705 (+1076434us) raft_consensus.cc:1683] finished
0831 07:59:06.060726 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:06.060835 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":42148,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":6046,"thread_start_us":1371198,"threads_started":1,"wal-append.queue_time_us":1791594}
I20250831 07:59:06.025640 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1130) took 2672 ms. Trace:
W20250831 07:59:05.899171 32435 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.060s user 0.001s sys 0.025s
W20250831 07:59:06.069205 32435 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.230s user 0.003s sys 0.030s
W20250831 07:59:06.069553 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 1.23017e+06us
I20250831 07:59:06.069111 31733 rpcz_store.cc:276] 0831 07:59:03.352977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.734722 (+381745us) service_pool.cc:225] Handling call
0831 07:59:03.789983 (+ 55261us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:03.790174 (+ 191us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 07:59:03.826349 (+ 36175us) log.cc:844] Serialized 14 byte log entry
0831 07:59:06.025302 (+2198953us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:06.025363 (+ 61us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:06.025404 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:06.025469 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:06.025588 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":35944,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":223944,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":152,"thread_start_us":2186577,"threads_started":2,"wal-append.queue_time_us":1660025}
I20250831 07:59:06.073766 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:06.031690 31875 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.006s sys 0.010s
W20250831 07:59:05.959138 1084 consensus_meta.cc:287] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.717s user 0.009s sys 0.002s
W20250831 07:59:05.805830 32462 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.024s sys 0.079s
W20250831 07:59:05.944588 1092 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.243s user 0.004s sys 0.069s
I20250831 07:59:06.008853 31878 rpcz_store.cc:276] 0831 07:59:03.403696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.944150 (+540454us) service_pool.cc:225] Handling call
0831 07:59:04.004561 (+ 60411us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:04.020949 (+ 16388us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:04.020977 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:04.021546 (+ 569us) log.cc:844] Serialized 26 byte log entry
0831 07:59:05.490374 (+1468828us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:05.490437 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:05.490540 (+ 103us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:05.831156 (+340616us) raft_consensus.cc:1683] finished
0831 07:59:05.831185 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:05.831380 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15899,"raft.queue_time_us":16055,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"thread_start_us":1467183,"threads_started":1,"wal-append.queue_time_us":1807105}
W20250831 07:59:06.023042 1032 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.112s user 0.023s sys 0.022s
I20250831 07:59:05.953377 914 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:05.953948 1110 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:06.006158 995 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:06.005273 1112 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:06.083786 1084 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:06.051151 1188 consensus_queue.cc:1035] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:59:06.052942 1234 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:06.028628 1007 consensus_meta.cc:287] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.864s user 0.010s sys 0.002s
I20250831 07:59:06.051725 1189 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:06.053299 32471 raft_consensus.cc:1238] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:06.053735 1230 consensus_queue.cc:1035] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:59:06.053947 1187 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:05.790220 1142 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:06.102783 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:05.703028 938 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:06.055217 1099 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:06.056061 898 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:06.008323 31726 rpcz_store.cc:276] 0831 07:59:03.354315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.999771 (+645456us) service_pool.cc:225] Handling call
0831 07:59:04.001284 (+ 1513us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:04.001744 (+ 460us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:04.001771 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:04.002386 (+ 615us) log.cc:844] Serialized 26 byte log entry
0831 07:59:05.262832 (+1260446us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:05.273817 (+ 10985us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:05.277920 (+ 4103us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:05.682144 (+404224us) raft_consensus.cc:1683] finished
0831 07:59:05.682173 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:05.682327 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1241753,"threads_started":1,"wal-append.queue_time_us":1572145}
I20250831 07:59:05.789758 954 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:06.056442 1231 consensus_queue.cc:1035] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20250831 07:59:06.056594 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:05.712337 1118 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:05.941968 1191 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:06.033169 1100 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:06.005723 1147 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:05.880237 1116 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:05.922761 1220 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:05.687887 909 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.771s user 0.000s sys 0.021s
I20250831 07:59:06.063613 31699 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:06.033852 31698 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:06.077113 31875 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.591s user 0.006s sys 0.010s
W20250831 07:59:06.078914 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:06.079176 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 59299.8us
I20250831 07:59:06.079332 1240 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:06.031479 1181 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 07:59:06.048784 1044 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.045s user 0.013s sys 0.027s
I20250831 07:59:06.087406 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:06.080931 32462 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.007s user 0.024s sys 0.082s
I20250831 07:59:06.026482 31727 rpcz_store.cc:276] 0831 07:59:03.353296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.825219 (+471923us) service_pool.cc:225] Handling call
0831 07:59:03.825682 (+ 463us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:03.825916 (+ 234us) raft_consensus.cc:1532] Early marking committed up to index 5
0831 07:59:05.470226 (+1644310us) log.cc:844] Serialized 14 byte log entry
0831 07:59:05.947074 (+476848us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:05.947150 (+ 76us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:05.947210 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:05.947313 (+ 103us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:05.947472 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31903,"prepare.queue_time_us":97499,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":85,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":92,"thread_start_us":2058611,"threads_started":2}
I20250831 07:59:06.067469 1232 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:06.097203 1234 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:06.054912 1222 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:06.145941 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 53448.2us
I20250831 07:59:06.106365 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:06.158179 1084 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:06.108577 1007 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:06.157861 1112 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:06.188390 31848 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:06.157727 978 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:06.171568 1210 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:06.189311 1231 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:06.111348 1018 consensus_queue.cc:260] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:06.189653 1100 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:06.153918 1219 log.cc:927] Time spent T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
I20250831 07:59:06.139873 1189 consensus_queue.cc:1046] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20250831 07:59:06.190003 1240 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:06.108253 1230 consensus_queue.cc:1230] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 07:59:06.181284 1180 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:06.191560 1147 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:06.192636 1220 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:06.189404 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 125884us
W20250831 07:59:06.200166 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 166406us
I20250831 07:59:06.205922 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1141) took 3300 ms. Trace:
I20250831 07:59:06.226733 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:06.226863 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:06.211951 1181 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:06.210310 1116 raft_consensus.cc:2802] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:06.228505 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:06.229523 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:06.121121 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:06.234114 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:06.237763 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:06.243145 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:06.243657 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:06.255306 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:06.255944 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:06.260861 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:06.256280 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:06.244688 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:06.250540 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:06.256582 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:06.235070 1002 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:06.272416 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:06.273135 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:06.273519 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:06.232271 940 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:06.229653 1243 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:06.244627 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:06.230839 1232 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:06.273837 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:06.551349 1234 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:06.551441 31732 raft_consensus.cc:2391] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:06.552155 1142 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:06.552738 1219 log.cc:927] Time spent T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.154s user 0.000s sys 0.000s
I20250831 07:59:06.552894 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:06.553603 31732 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:06.554284 1240 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:06.554425 1249 raft_consensus.cc:3053] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:06.554567 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:06.554898 1249 raft_consensus.cc:738] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:06.556103 31882 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
W20250831 07:59:06.556502 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 71371.1us
I20250831 07:59:06.557008 31698 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:06.558161 31698 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:06.560307 31802 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:06.567606 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:06.564498 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:06.560657 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:06.531287 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:06.571067 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:06.563033 31800 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:59:06.562827 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 75688.6us
I20250831 07:59:06.472746 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:06.572650 31725 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:06.564569 31804 raft_consensus.cc:2408] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:06.565006 31729 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
W20250831 07:59:06.506598 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 269500us
I20250831 07:59:06.574260 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:06.575222 31804 raft_consensus.cc:2391] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:06.575603 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:06.575953 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:06.576642 31804 raft_consensus.cc:2391] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:06.577261 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:06.566385 31879 raft_consensus.cc:2408] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:06.313308 31803 raft_consensus.cc:1238] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 07:59:06.563385 31881 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:06.562783 31805 raft_consensus.cc:2391] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:06.240495 1231 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:06.404497 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 173561us
I20250831 07:59:06.563550 31733 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:06.577419 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:06.565289 31699 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:06.263868 995 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election won for term 4
W20250831 07:59:06.335848 1196 log.cc:927] Time spent T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20250831 07:59:06.569494 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:06.725361 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 109053us
I20250831 07:59:06.561120 31875 rpcz_store.cc:276] 0831 07:59:02.905558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:03.255603 (+350045us) service_pool.cc:225] Handling call
0831 07:59:03.256114 (+ 511us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:04.543286 (+1287172us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:04.543331 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:04.628704 (+ 85373us) log.cc:844] Serialized 26 byte log entry
0831 07:59:05.458559 (+829855us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:06.188263 (+729704us) log.cc:844] Serialized 14 byte log entry
0831 07:59:06.205476 (+ 17213us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:06.205586 (+ 110us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:06.205652 (+ 66us) raft_consensus.cc:1683] finished
0831 07:59:06.205672 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:06.205848 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36490,"prepare.queue_time_us":542886,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":79238,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":16908,"thread_start_us":2755171,"threads_started":4,"wal-append.queue_time_us":1039187}
I20250831 07:59:06.823280 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:06.239197 1210 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:06.824522 31732 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:06.588764 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:06.830790 31734 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
W20250831 07:59:06.830838 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 265658us
I20250831 07:59:06.614794 31728 raft_consensus.cc:2391] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:06.824505 31698 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:06.658905 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:06.835358 31801 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:06.659768 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:06.653577 31806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1567) took 2182 ms (client timeout 2631 ms). Trace:
I20250831 07:59:06.824970 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:06.827207 31882 raft_consensus.cc:2391] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:06.837041 31875 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:06.825209 32434 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:06.829646 31729 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:06.681659 31881 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:06.274257 1018 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:06.315881 32462 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 305) took 1528 ms. Trace:
I20250831 07:59:06.569832 1181 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:06.723009 32465 raft_consensus.cc:2391] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
W20250831 07:59:06.836750 31806 rpcz_store.cc:269] 0831 07:59:04.470827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:04.546989 (+ 76162us) service_pool.cc:225] Handling call
0831 07:59:06.653288 (+2106299us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54000398d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:06.653470 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3118724992}
I20250831 07:59:06.507220 1067 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:06.842597 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:06.861194 31875 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:06.824008 1007 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:06.344671 1008 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:06.312601 1188 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:06.570063 1234 raft_consensus.cc:2802] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:06.846338 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:06.346848 1123 raft_consensus.cc:3053] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:06.473304 1032 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:06.571758 1197 raft_consensus.cc:2802] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:06.650681 954 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:06.402287 1110 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:06.569947 1118 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:06.833026 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:06.580298 1002 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:06.929872 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:06.930959 31879 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:06.821900 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:06.839632 1233 log.cc:927] Time spent T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.116s user 0.001s sys 0.000s
I20250831 07:59:06.862457 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:06.400137 1189 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:06.234727 1222 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:06.720645 1256 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:06.820811 993 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:06.768071 1228 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.025s sys 0.037s
I20250831 07:59:06.829355 1231 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:06.946144 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:06.852684 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:06.853828 1230 consensus_queue.cc:1035] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.074s
I20250831 07:59:06.578711 940 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:06.910877 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:06.907697 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:06.917819 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:06.919387 31733 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:06.920296 31729 raft_consensus.cc:1238] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 07:59:06.847833 1044 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:06.854028 995 raft_consensus.cc:695] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:06.767800 1142 consensus_queue.cc:1035] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.107s
I20250831 07:59:06.932943 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:06.860306 1100 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:06.936180 31875 raft_consensus.cc:2391] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:06.919173 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:07.013166 31848 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:07.014348 31848 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:06.944542 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:06.653119 1249 consensus_queue.cc:260] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:06.945299 1197 raft_consensus.cc:695] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:06.940037 1189 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:06.616003 1116 raft_consensus.cc:695] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:06.952351 31882 raft_consensus.cc:2374] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2. Re-sending same reply.
I20250831 07:59:06.578476 1243 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:06.953760 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:06.921567 1123 raft_consensus.cc:738] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:06.928267 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:06.995716 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:06.998742 31803 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:07.100258 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.101284 31879 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:06.999750 31805 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:06.934592 1032 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:06.961704 31880 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:06.907400 1067 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:07.023676 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.032600 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:07.021629 31877 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.759s user 0.035s sys 0.054s
W20250831 07:59:07.112067 31877 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.849s user 0.039s sys 0.073s
W20250831 07:59:07.036859 31874 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.017s sys 0.050s
W20250831 07:59:06.953066 1228 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s user 0.027s sys 0.040s
I20250831 07:59:06.923843 1188 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:06.954146 909 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election won for term 5
W20250831 07:59:06.925642 938 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.011s sys 0.006s
I20250831 07:59:07.107244 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.110091 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.004459 1044 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:06.954960 1256 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:07.419571 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:07.271209 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 446047us
I20250831 07:59:07.475999 31729 raft_consensus.cc:2391] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:07.143836 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:07.003124 978 raft_consensus.cc:2747] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:07.180665 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:07.431947 31874 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.128s user 0.030s sys 0.075s
W20250831 07:59:07.212196 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 385598us
I20250831 07:59:07.113556 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.228298 31801 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:06.948550 32466 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:59:07.430053 938 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.183s user 0.013s sys 0.024s
W20250831 07:59:07.159482 31849 leader_election.cc:336] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.579s (SENT)
I20250831 07:59:07.512307 31849 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:07.512581 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:07.158413 1251 log.cc:927] Time spent T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.435s user 0.001s sys 0.000s
I20250831 07:59:06.947186 1084 raft_consensus.cc:2747] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:07.168777 1232 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:07.481952 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:07.529415 31882 raft_consensus.cc:2391] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:07.472740 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47886 (request call id 359) took 1385 ms. Trace:
I20250831 07:59:07.465890 31879 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:07.096448 1118 raft_consensus.cc:2802] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:07.358325 1240 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:07.499451 31848 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:07.432468 31699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.000s sys 0.009s
W20250831 07:59:07.534482 31699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.696s user 0.003s sys 0.012s
I20250831 07:59:07.534765 31848 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:07.534888 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 700794us
I20250831 07:59:07.534933 31875 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
W20250831 07:59:07.234223 1252 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
I20250831 07:59:07.517691 32434 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65
W20250831 07:59:07.466543 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:07.257148 1220 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:07.478001 909 raft_consensus.cc:695] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:07.419620 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:07.520597 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:07.522257 32436 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.677s user 0.016s sys 0.029s
I20250831 07:59:07.522521 31877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1161) took 1055 ms. Trace:
I20250831 07:59:07.545846 31877 rpcz_store.cc:276] 0831 07:59:06.467150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:07.512283 (+1045133us) service_pool.cc:225] Handling call
0831 07:59:07.522457 (+ 10174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:07.123741 1249 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:07.433918 1002 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:07.522257 31728 raft_consensus.cc:2374] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4. Re-sending same reply.
I20250831 07:59:07.525533 31732 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d"
dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:48724
I20250831 07:59:07.527424 1142 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:07.499626 31801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1586) took 1250 ms (client timeout 1593 ms). Trace:
I20250831 07:59:07.528301 1032 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:07.234459 1116 consensus_queue.cc:237] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:06.848146 32462 rpcz_store.cc:276] 0831 07:59:04.787629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:04.789590 (+ 1961us) service_pool.cc:225] Handling call
0831 07:59:04.814355 (+ 24765us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:04.887898 (+ 73543us) raft_consensus.cc:1532] Early marking committed up to index 5
0831 07:59:04.993805 (+105907us) log.cc:844] Serialized 14 byte log entry
0831 07:59:06.232747 (+1238942us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:06.234605 (+ 1858us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:06.246836 (+ 12231us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:06.263478 (+ 16642us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:06.306841 (+ 43363us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3153,"prepare.queue_time_us":88268,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":59,"raft.queue_time_us":6820,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":13987,"thread_start_us":1143351,"threads_started":1,"wal-append.queue_time_us":1000478,"wal-append.run_cpu_time_us":13895,"wal-append.run_wall_time_us":1036639}
I20250831 07:59:07.507771 31802 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:07.123515 1241 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:07.466660 1210 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:07.531641 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1157) took 1231 ms. Trace:
I20250831 07:59:07.176662 995 consensus_queue.cc:237] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:07.519328 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 359865us
I20250831 07:59:07.519378 1110 raft_consensus.cc:2747] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:07.141989 1239 consensus_queue.cc:1046] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.391s
I20250831 07:59:07.236313 1123 consensus_queue.cc:260] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:07.528313 1222 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:07.140159 993 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:07.547608 31805 raft_consensus.cc:2391] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:59:07.547739 32436 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.702s user 0.017s sys 0.029s
I20250831 07:59:07.548971 31732 raft_consensus.cc:604] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:07.594030 31732 raft_consensus.cc:612] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250831 07:59:07.552127 31801 rpcz_store.cc:269] 0831 07:59:06.249273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.570054 (+320781us) service_pool.cc:225] Handling call
0831 07:59:07.499542 (+929488us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:07.553680 1189 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:07.602532 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:07.603622 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:07.603981 31802 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:07.234170 1197 consensus_queue.cc:237] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:07.538838 1232 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:07.528522 1084 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:07.535717 1118 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:59:07.564240 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 46628.7us
I20250831 07:59:07.565200 1243 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:07.533061 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1179) took 1061 ms. Trace:
I20250831 07:59:07.619850 31879 rpcz_store.cc:276] 0831 07:59:06.471771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:07.101861 (+630090us) service_pool.cc:225] Handling call
0831 07:59:07.532999 (+431138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:07.544584 1013 raft_consensus.cc:2802] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250831 07:59:07.548982 1257 log.cc:927] Time spent T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.451s user 0.001s sys 0.001s
I20250831 07:59:07.546753 1220 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:07.595261 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:07.553648 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.630008 31877 raft_consensus.cc:2391] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:07.596827 31848 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:07.597810 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 761141us
I20250831 07:59:07.604943 31878 raft_consensus.cc:2391] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:07.554976 940 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:07.552838 31726 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
W20250831 07:59:07.610646 31725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.017s sys 0.035s
W20250831 07:59:07.611035 31698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.772s user 0.008s sys 0.044s
I20250831 07:59:07.565613 31882 rpcz_store.cc:276] 0831 07:59:06.301115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:07.475053 (+1173938us) service_pool.cc:225] Handling call
0831 07:59:07.531558 (+ 56505us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:07.586866 1092 raft_consensus.cc:2802] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:07.592168 1067 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:07.592854 1260 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:07.663484 1260 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:07.665778 1260 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:07.545069 1240 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:07.668000 1240 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:07.636564 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.668937 31877 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:07.669572 31801 raft_consensus.cc:1238] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 07:59:07.669728 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.670614 31877 raft_consensus.cc:2391] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
W20250831 07:59:07.672379 1272 log.cc:927] Time spent T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20250831 07:59:07.607722 1123 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:07.676187 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:07.676787 32434 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:07.677331 1270 log.cc:927] Time spent T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20250831 07:59:07.677956 32434 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:07.678644 1262 log.cc:927] Time spent T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.180s user 0.000s sys 0.001s
I20250831 07:59:07.564895 1142 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:07.655876 31725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s user 0.018s sys 0.035s
W20250831 07:59:07.657449 31698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.818s user 0.008s sys 0.044s
W20250831 07:59:07.679770 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 855311us
I20250831 07:59:07.680198 32468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 316) took 1581 ms. Trace:
I20250831 07:59:07.680425 31698 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:07.654736 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.532184 32469 rpcz_store.cc:276] 0831 07:59:06.081583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.084892 (+ 3309us) service_pool.cc:225] Handling call
0831 07:59:07.021078 (+936186us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e5ce 000000000035a3c9 00007fdcecee5300 00007fdcecfabec4 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:07.152522 (+131444us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:07.174624 (+ 22102us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:07.175730 (+ 1106us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:07.175808 (+ 78us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:07.229492 (+ 53684us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:07.271459 (+ 41967us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:07.466780 (+195321us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26177280}
I20250831 07:59:07.681435 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1167) took 1212 ms. Trace:
I20250831 07:59:07.681576 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:07.681787 31879 rpcz_store.cc:276] 0831 07:59:06.468790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:07.620860 (+1152070us) service_pool.cc:225] Handling call
0831 07:59:07.681383 (+ 60523us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:07.682183 31806 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:59:07.682670 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.618445 1118 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:07.683053 32464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47886 (request call id 361) took 1596 ms. Trace:
I20250831 07:59:07.620719 1275 consensus_queue.cc:1046] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20250831 07:59:07.623426 1013 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:07.593019 1241 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:07.684746 1013 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:07.686887 1241 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:07.622767 1084 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:07.662340 1092 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:07.680730 32468 rpcz_store.cc:276] 0831 07:59:06.098772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.098911 (+ 139us) service_pool.cc:225] Handling call
0831 07:59:06.266998 (+168087us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:06.267200 (+ 202us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:06.274880 (+ 7680us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:06.274926 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:06.300794 (+ 25868us) log.cc:844] Serialized 26 byte log entry
0831 07:59:06.995059 (+694265us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:06.995223 (+ 164us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:06.995291 (+ 68us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:07.679793 (+684502us) raft_consensus.cc:1683] finished
0831 07:59:07.679847 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:07.680109 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12012,"raft.queue_time_us":208814,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":48242,"spinlock_wait_cycles":2791168,"thread_start_us":473824,"threads_started":1,"wal-append.queue_time_us":982622}
I20250831 07:59:07.693565 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:07.690351 31879 raft_consensus.cc:2391] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:07.671484 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:07.697109 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:07.705371 31808 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:07.676831 32462 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:07.711059 1084 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:07.680554 1142 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:07.711601 1223 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:07.713459 1223 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:07.713910 1223 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:07.713680 1118 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:07.715201 1013 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:07.691514 31848 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:07.715958 1118 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:07.624230 1232 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:07.704005 1256 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:07.717470 1274 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:07.717628 1232 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:07.717991 1274 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:07.719005 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1157) took 1031 ms. Trace:
W20250831 07:59:07.720017 1266 log.cc:927] Time spent T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.254s user 0.001s sys 0.000s
I20250831 07:59:07.720139 1239 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:07.719398 31725 rpcz_store.cc:276] 0831 07:59:06.689060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.689351 (+ 291us) service_pool.cc:225] Handling call
0831 07:59:06.797060 (+107709us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:06.797291 (+ 231us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 07:59:07.033675 (+236384us) log.cc:844] Serialized 14 byte log entry
0831 07:59:07.718649 (+684974us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:07.718699 (+ 50us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:07.718744 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:07.718817 (+ 73us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:07.718949 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39139,"prepare.queue_time_us":529877,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":37,"thread_start_us":840159,"threads_started":2}
I20250831 07:59:07.720635 1239 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:07.721498 31698 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:07.718035 31848 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:07.629222 1220 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:07.704211 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.734030 31882 raft_consensus.cc:2391] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:07.656597 993 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:07.736891 31698 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:07.745664 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:07.746452 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:07.724392 1018 consensus_meta.cc:287] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.865s user 0.005s sys 0.001s
I20250831 07:59:07.760895 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:07.738126 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.762050 31882 raft_consensus.cc:2391] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:07.738891 1232 raft_consensus.cc:2747] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:07.739060 1275 raft_consensus.cc:3053] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:07.764160 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:07.739177 1243 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:07.739775 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.740717 954 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:07.736125 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:07.772330 31806 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:07.742672 1013 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:07.744129 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:07.589653 909 consensus_queue.cc:237] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:07.687460 1241 consensus_queue.cc:1046] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20250831 07:59:07.694361 1246 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:07.721730 1092 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:07.765880 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1174) took 1295 ms. Trace:
I20250831 07:59:07.812757 31882 rpcz_store.cc:276] 0831 07:59:06.470538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:07.735239 (+1264701us) service_pool.cc:225] Handling call
0831 07:59:07.765804 (+ 30565us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:07.743811 1181 consensus_meta.cc:287] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.869s user 0.009s sys 0.001s
I20250831 07:59:07.818389 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.769697 31807 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:07.726897 1044 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:07.727190 1240 raft_consensus.cc:2802] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:07.772094 954 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:07.775740 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:07.834523 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:07.777220 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:07.783423 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 256274us
W20250831 07:59:07.784873 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 196797us
I20250831 07:59:07.761114 1220 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:07.767769 993 raft_consensus.cc:695] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:07.763532 1275 raft_consensus.cc:738] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:07.819020 31879 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:07.820911 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:07.811211 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:07.769038 1263 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:07.771457 1243 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:07.772373 1084 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:07.774243 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:07.829752 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.832131 1264 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:59:07.775842 1274 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:07.833585 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:07.778142 1188 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:07.816210 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:07.839481 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:07.840479 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:07.683583 32464 rpcz_store.cc:276] 0831 07:59:06.086336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.089267 (+ 2931us) service_pool.cc:225] Handling call
0831 07:59:07.682112 (+1592845us) spinlock_profiling.cc:243] Waited 8.66 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e5ce 000000000035a3c9 00007fdcecee5300 00007fdcecfabec4 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:07.682320 (+ 208us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:07.682575 (+ 255us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:07.682624 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:07.682690 (+ 66us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:07.682749 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:07.682830 (+ 81us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:07.682971 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19916160}
I20250831 07:59:07.869287 31732 raft_consensus.cc:2466] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:07.824716 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:07.835808 954 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:07.825177 1232 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:07.855933 1247 log.cc:927] Time spent T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 1.292s user 0.004s sys 0.000s
I20250831 07:59:07.834597 31806 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:07.830030 1181 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:07.847177 1180 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:07.967799 31882 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:07.968518 31848 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:07.969185 31725 raft_consensus.cc:2374] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4. Re-sending same reply.
I20250831 07:59:07.971127 31801 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:07.901859 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:07.923484 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:07.878213 31775 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:07.975365 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:07.821162 1246 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:07.957147 1270 log.cc:927] Time spent T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20250831 07:59:07.997435 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1589) took 1747 ms. Trace:
I20250831 07:59:07.997864 31800 rpcz_store.cc:276] 0831 07:59:06.250890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.577829 (+326939us) service_pool.cc:225] Handling call
0831 07:59:06.844396 (+266567us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:07.102288 (+257892us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:07.102334 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:07.102394 (+ 60us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:07.102434 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:07.997193 (+894759us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b54000ce9d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:07.997241 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:07.997372 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":592501120}
I20250831 07:59:07.980777 1241 raft_consensus.cc:2802] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:07.999936 1084 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:08.000461 1241 raft_consensus.cc:695] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:08.000787 1084 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:08.001528 1241 consensus_queue.cc:237] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:07.980239 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:08.003310 31730 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
W20250831 07:59:08.003684 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 125543us
W20250831 07:59:08.000681 31807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1581) took 1758 ms (client timeout 1748 ms). Trace:
I20250831 07:59:08.004693 31775 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:08.005201 31807 rpcz_store.cc:269] 0831 07:59:06.242114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.242232 (+ 118us) service_pool.cc:225] Handling call
0831 07:59:08.000519 (+1758287us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54003a07d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:08.000616 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2770737024}
I20250831 07:59:08.005890 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:07.876008 32469 raft_consensus.cc:1273] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:07.981381 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:08.023872 31806 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
W20250831 07:59:07.982928 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:07.983152 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:08.025260 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:07.987157 993 consensus_queue.cc:237] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:07.987886 1239 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:07.970139 1231 consensus_meta.cc:287] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.967s user 0.011s sys 0.000s
I20250831 07:59:07.992182 31876 raft_consensus.cc:2366] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4.
I20250831 07:59:07.992684 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:07.993607 1181 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:07.958155 1191 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:07.839807 938 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.177s
I20250831 07:59:07.994227 1246 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:07.900163 1263 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:07.975636 1275 consensus_queue.cc:260] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:07.995756 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1149) took 1681 ms. Trace:
W20250831 07:59:08.015951 31849 leader_election.cc:336] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.741s (SENT)
I20250831 07:59:08.016515 31801 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:07.979526 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:07.993808 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:08.168444 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:07.974237 1243 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:08.266705 31875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1171) took 1796 ms (client timeout 2024 ms). Trace:
W20250831 07:59:08.149971 31878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 139) took 1897 ms (client timeout 2037 ms). Trace:
I20250831 07:59:08.180747 31698 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:08.026100 31730 raft_consensus.cc:2374] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3. Re-sending same reply.
I20250831 07:59:08.278951 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:08.271788 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:08.279228 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 403668us
I20250831 07:59:08.031745 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:08.281849 31875 rpcz_store.cc:269] 0831 07:59:06.469797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:07.023356 (+553559us) service_pool.cc:225] Handling call
0831 07:59:08.266474 (+1243118us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b54000f7b54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:08.266623 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":973100928}
I20250831 07:59:08.281731 31806 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:08.267264 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1580) took 2025 ms. Trace:
I20250831 07:59:08.286804 31809 rpcz_store.cc:276] 0831 07:59:06.241835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.243578 (+ 1743us) service_pool.cc:225] Handling call
0831 07:59:06.243856 (+ 278us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:06.246553 (+ 2697us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:06.246598 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:06.246656 (+ 58us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:06.246698 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:08.266987 (+2020289us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b54000852d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:08.267047 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:08.267192 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5689216}
W20250831 07:59:08.289569 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 273645us
I20250831 07:59:08.283260 31882 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:59:07.981154 1274 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:08.280560 31878 rpcz_store.cc:269] 0831 07:59:06.253091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.253241 (+ 150us) service_pool.cc:225] Handling call
0831 07:59:06.995577 (+742336us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e57c 00007fdce505e13b 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:08.149697 (+1154120us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b540001e0d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:08.149889 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1116140544}
I20250831 07:59:08.300230 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:08.055549 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:08.301668 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:08.302114 31883 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:08.301666 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:08.302618 31883 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1155) took 2045 ms (client timeout 1541 ms). Trace:
I20250831 07:59:08.303282 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:08.303488 31883 rpcz_store.cc:269] 0831 07:59:06.257116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.552162 (+295046us) service_pool.cc:225] Handling call
0831 07:59:08.055352 (+1503190us) spinlock_profiling.cc:243] Waited 7.32 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:08.302554 (+247202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16829696}
I20250831 07:59:08.304592 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:08.306214 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:08.310748 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 130122us
W20250831 07:59:08.030956 32466 consensus_meta.cc:287] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.501s user 0.009s sys 0.000s
W20250831 07:59:08.314172 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 1.82629e+06us
I20250831 07:59:08.314426 31806 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:08.316203 31883 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:08.306051 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:08.086432 1188 raft_consensus.cc:2802] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:08.321226 31847 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:07.977761 1180 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:08.086203 1249 consensus_meta.cc:287] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.534s user 0.006s sys 0.003s
W20250831 07:59:08.031215 1273 log.cc:927] Time spent T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.197s user 0.001s sys 0.000s
I20250831 07:59:08.045962 31775 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:08.364883 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 318980us
W20250831 07:59:08.290706 31849 leader_election.cc:336] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.781s (SENT)
I20250831 07:59:08.145092 1283 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:08.430401 31881 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:08.024145 1261 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:08.262209 1084 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:08.052520 1223 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:07.974958 1232 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:08.338476 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:08.299451 1281 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:08.271368 31727 rpcz_store.cc:276] 0831 07:59:06.314193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.559806 (+245613us) service_pool.cc:225] Handling call
0831 07:59:06.621902 (+ 62096us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:07.159202 (+537300us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:07.159246 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:07.159329 (+ 83us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:07.159398 (+ 69us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:07.995529 (+836131us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b54000c21d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:07.995574 (+ 45us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:07.995692 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1235101056}
W20250831 07:59:08.182598 32436 leader_election.cc:336] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.190s (SENT)
I20250831 07:59:08.055081 1231 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:08.366567 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 75893.2us
I20250831 07:59:08.371894 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:08.429606 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 63783.1us
I20250831 07:59:08.319964 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:08.189803 1092 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:08.367781 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 96275.7us
W20250831 07:59:09.194847 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:09.199131 31732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1613) took 1349 ms (client timeout 1762 ms). Trace:
W20250831 07:59:09.199901 31732 rpcz_store.cc:269] 0831 07:59:07.849171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:08.300361 (+451190us) service_pool.cc:225] Handling call
0831 07:59:09.198831 (+898470us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:08.440423 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:09.202250 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 144) took 1186 ms. Trace:
I20250831 07:59:08.583550 31726 raft_consensus.cc:2391] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:09.203109 31733 rpcz_store.cc:276] 0831 07:59:08.016038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:08.440084 (+424046us) service_pool.cc:225] Handling call
0831 07:59:09.201994 (+761910us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:09.202593 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:08.805461 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1163) took 2337 ms. Trace:
I20250831 07:59:09.244930 31882 rpcz_store.cc:276] 0831 07:59:06.467737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:08.305660 (+1837923us) service_pool.cc:225] Handling call
0831 07:59:08.313765 (+ 8105us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:08.313991 (+ 226us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 07:59:08.363412 (+ 49421us) log.cc:844] Serialized 14 byte log entry
0831 07:59:08.804959 (+441547us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:08.805038 (+ 79us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:08.805107 (+ 69us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:08.805196 (+ 89us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:08.805371 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":115407,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":16214,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":104,"thread_start_us":441083,"threads_started":1}
I20250831 07:59:09.247144 1317 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:09.248720 1317 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:09.250236 31882 raft_consensus.cc:1238] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:09.251247 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1170) took 2781 ms. Trace:
I20250831 07:59:09.251643 31882 rpcz_store.cc:276] 0831 07:59:06.469556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.248828 (+2779272us) service_pool.cc:225] Handling call
0831 07:59:09.249812 (+ 984us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:09.250837 (+ 1025us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:09.251110 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:09.255475 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1158) took 2910 ms. Trace:
W20250831 07:59:08.551043 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:09.256084 31875 rpcz_store.cc:276] 0831 07:59:06.344357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:08.304529 (+1960172us) service_pool.cc:225] Handling call
0831 07:59:08.304849 (+ 320us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:08.338883 (+ 34034us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:08.338928 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:08.338995 (+ 67us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:08.339066 (+ 71us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:08.339153 (+ 87us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:09.254103 (+914950us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44902,"raft.queue_time_us":59028,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":45169}
I20250831 07:59:09.258951 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:09.271847 31848 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:09.272097 32434 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.000s sys 0.009s
I20250831 07:59:09.272241 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1569) took 1092 ms. Trace:
W20250831 07:59:09.272835 32434 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.536s user 0.000s sys 0.009s
I20250831 07:59:09.272866 31878 rpcz_store.cc:276] 0831 07:59:08.179761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:08.302614 (+122853us) service_pool.cc:225] Handling call
0831 07:59:09.271969 (+969355us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:09.273844 31848 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:09.270291 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1621) took 1399 ms. Trace:
I20250831 07:59:09.278582 31733 rpcz_store.cc:276] 0831 07:59:07.869509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.258488 (+1388979us) service_pool.cc:225] Handling call
0831 07:59:09.268762 (+ 10274us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:09.289528 31725 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:09.290271 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1599) took 1494 ms. Trace:
I20250831 07:59:09.290169 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:09.290402 31849 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:09.290729 31725 rpcz_store.cc:276] 0831 07:59:07.795431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:08.314912 (+519481us) service_pool.cc:225] Handling call
0831 07:59:09.290188 (+975276us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:09.291236 31727 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:09.300559 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 147) took 1008 ms. Trace:
I20250831 07:59:09.302736 31727 rpcz_store.cc:276] 0831 07:59:08.283283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.289618 (+1006335us) service_pool.cc:225] Handling call
0831 07:59:09.291782 (+ 2164us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:09.303102 32435 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.003s sys 0.007s
W20250831 07:59:09.303676 32435 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.003s sys 0.007s
I20250831 07:59:09.304044 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:09.305419 31847 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.000s sys 0.011s
W20250831 07:59:09.306035 31847 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.825s user 0.000s sys 0.011s
W20250831 07:59:09.306679 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 985483us
I20250831 07:59:09.308734 31879 raft_consensus.cc:1238] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:08.325031 32466 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:09.309532 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1162) took 2842 ms. Trace:
W20250831 07:59:09.309790 32466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47886 (request call id 360) took 3225 ms (client timeout 2243 ms). Trace:
I20250831 07:59:09.309991 31879 rpcz_store.cc:276] 0831 07:59:06.467441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:08.305504 (+1838063us) service_pool.cc:225] Handling call
0831 07:59:09.308054 (+1002550us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:09.309244 (+ 1190us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:09.309465 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:09.310292 32466 rpcz_store.cc:269] 0831 07:59:06.084100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.086096 (+ 1996us) service_pool.cc:225] Handling call
0831 07:59:06.821726 (+735630us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:09.309514 (+2487788us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37417728}
I20250831 07:59:08.687374 32469 raft_consensus.cc:1238] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250831 07:59:09.321990 1318 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:09.322597 909 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.002s sys 0.007s
W20250831 07:59:09.323181 909 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s user 0.002s sys 0.007s
I20250831 07:59:09.324693 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:09.326267 954 consensus_meta.cc:287] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.346s user 0.012s sys 0.004s
I20250831 07:59:09.327852 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:09.309351 1316 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:09.329162 1316 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:09.330054 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:09.327037 954 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:09.472086 1309 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:09.473137 1309 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:09.292064 1317 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:09.272141 31726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1604) took 1394 ms (client timeout 1748 ms). Trace:
I20250831 07:59:08.431089 1246 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:09.492945 1246 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:08.473448 1261 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:09.496587 1261 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:09.497351 1013 raft_consensus.cc:2802] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:08.552932 1283 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:09.488195 31726 rpcz_store.cc:269] 0831 07:59:07.815668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:07.993534 (+177866us) service_pool.cc:225] Handling call
0831 07:59:09.203999 (+1210465us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:08.935159 1234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.981s user 0.040s sys 0.096s
W20250831 07:59:09.500452 1234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.548s user 0.048s sys 0.098s
I20250831 07:59:09.323125 1318 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:09.509147 31732 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:59:09.504303 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:09.504762 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:08.317330 1116 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:09.304322 1322 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:08.753019 1301 log.cc:927] Time spent T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.171s user 0.000s sys 0.000s
W20250831 07:59:08.836648 1220 consensus_meta.cc:287] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.966s user 0.010s sys 0.001s
I20250831 07:59:09.501299 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:08.771783 1300 log.cc:927] Time spent T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.267s user 0.000s sys 0.001s
I20250831 07:59:09.500811 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:09.641402 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 336139us
I20250831 07:59:08.326372 1188 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:08.834820 1304 log.cc:927] Time spent T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20250831 07:59:08.979167 1297 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:08.499018 1249 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:09.505189 1317 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:09.701676 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:09.708141 31725 raft_consensus.cc:2391] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:09.643318 995 raft_consensus.cc:2802] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:09.650841 1309 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:08.440119 1279 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:08.442931 1191 raft_consensus.cc:2802] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:09.707747 1188 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:09.724206 1191 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:08.833822 1263 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:09.724772 1191 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:09.712975 1220 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:09.712932 1223 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:09.727574 31877 raft_consensus.cc:2391] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:09.727829 1310 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:09.658591 1013 raft_consensus.cc:695] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:09.719969 1249 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:09.730067 1249 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:09.729964 1013 consensus_queue.cc:237] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:09.731302 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:09.731990 1197 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:09.732926 1220 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:09.732733 1197 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:09.720396 31726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1603) took 1911 ms (client timeout 2138 ms). Trace:
W20250831 07:59:09.734951 31726 rpcz_store.cc:269] 0831 07:59:07.809372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.500160 (+1690788us) service_pool.cc:225] Handling call
0831 07:59:09.720310 (+220150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:09.735258 1197 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:09.710539 995 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:09.736121 995 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:09.500850 1234 raft_consensus.cc:695] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:09.712970 31698 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:09.721390 31733 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:09.722204 1297 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:09.722295 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:09.723203 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 433156us
I20250831 07:59:09.716121 1317 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:09.713022 1210 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250831 07:59:09.749541 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 441681us
W20250831 07:59:09.765224 31881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1168) took 3276 ms (client timeout 1763 ms). Trace:
W20250831 07:59:09.765821 31881 rpcz_store.cc:269] 0831 07:59:06.469044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.845998 (+376954us) service_pool.cc:225] Handling call
0831 07:59:09.736936 (+2890938us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b5400246654. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:09.745196 (+ 8260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3499452800}
I20250831 07:59:09.766595 32469 raft_consensus.cc:2391] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:09.723513 1279 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:09.770901 1116 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:09.781311 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 1.5985e+06us
W20250831 07:59:09.781744 31773 connection.cc:232] RPC call timeout handler was delayed by -1.6968284109999558s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2127s for a response
I20250831 07:59:09.726101 1240 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:08.315152 1274 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:09.736527 1002 raft_consensus.cc:3053] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Stepping down as leader of term 3
W20250831 07:59:09.799768 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:09.800562 31729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1615) took 1947 ms (client timeout 2119 ms). Trace:
W20250831 07:59:09.798616 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 357607us
I20250831 07:59:09.794476 1316 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:09.804538 31849 connection.cc:232] RPC call timeout handler was delayed by -1.4444284540001036s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2002s for a response
I20250831 07:59:09.800465 31848 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:09.854794 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:09.971585 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 146) took 2430 ms. Trace:
W20250831 07:59:09.803211 31729 rpcz_store.cc:269] 0831 07:59:07.849898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.730790 (+1880892us) service_pool.cc:225] Handling call
0831 07:59:09.796822 (+ 66032us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:09.896533 32434 leader_election.cc:336] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 2.037s (SENT)
W20250831 07:59:09.846174 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 247529us
W20250831 07:59:09.970598 31849 connection.cc:232] RPC call timeout handler was delayed by -1.4399038709999559s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19820000000000002s for a response
I20250831 07:59:09.862900 31727 raft_consensus.cc:2466] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:59:09.796428 1240 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:09.975394 31882 rpcz_store.cc:276] 0831 07:59:07.541013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.254884 (+1713871us) service_pool.cc:225] Handling call
0831 07:59:09.255853 (+ 969us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:09.257925 (+ 2072us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:09.257980 (+ 55us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:09.259048 (+ 1068us) log.cc:844] Serialized 26 byte log entry
0831 07:59:09.614945 (+355897us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:09.615002 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:09.615110 (+ 108us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:09.969456 (+354346us) raft_consensus.cc:1683] finished
0831 07:59:09.969478 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:09.971510 (+ 2032us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":355287,"threads_started":1,"wal-append.queue_time_us":484194}
I20250831 07:59:10.000887 32434 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:09.797266 1116 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:09.884474 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:09.880522 1190 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:09.853498 1283 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:09.873080 31847 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:09.878419 1188 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:10.013872 1313 log.cc:927] Time spent T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.187s user 0.000s sys 0.006s
I20250831 07:59:09.879683 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:09.801218 1002 raft_consensus.cc:738] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:09.964262 1329 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250831 07:59:10.014963 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 306211us
I20250831 07:59:09.879809 954 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:10.017011 31773 connection.cc:232] RPC call timeout handler was delayed by -1.4745284109999375s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23740000000000003s for a response
I20250831 07:59:09.872771 1232 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:09.803154 1246 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:09.877997 1092 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:10.022910 32467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47886 (request call id 370) took 2238 ms (client timeout 1763 ms). Trace:
I20250831 07:59:09.893451 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:10.021776 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 149) took 2479 ms. Trace:
I20250831 07:59:09.896778 1324 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:09.896790 1306 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:10.010466 1316 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:09.896574 1303 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:10.026420 31775 connection.cc:232] RPC call timeout handler was delayed by -1.4701287289999527s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.191s for a response
I20250831 07:59:10.028734 1044 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:10.031409 1142 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:10.029230 995 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:10.049314 1317 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:10.051223 31732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1624) took 2143 ms (client timeout 1930 ms). Trace:
W20250831 07:59:10.034009 1261 consensus_meta.cc:287] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.537s user 0.009s sys 0.005s
W20250831 07:59:10.051734 31732 rpcz_store.cc:269] 0831 07:59:07.908432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.201967 (+1293535us) service_pool.cc:225] Handling call
0831 07:59:10.051015 (+849048us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b540038d1d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:10.051142 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":585735552}
I20250831 07:59:10.052452 1261 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:10.053002 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:10.053175 31731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1144) took 3817 ms (client timeout 2766 ms). Trace:
I20250831 07:59:10.053481 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:10.054062 1331 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:10.054735 1261 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:10.035461 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 217878us
I20250831 07:59:10.027662 1329 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:10.036171 1312 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:10.036386 1231 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:10.037310 31726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1596) took 2242 ms (client timeout 2191 ms). Trace:
I20250831 07:59:09.898797 1249 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:10.029428 31876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1156) took 3771 ms (client timeout 1728 ms). Trace:
W20250831 07:59:10.038769 32467 rpcz_store.cc:269] 0831 07:59:07.784540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:07.784685 (+ 145us) service_pool.cc:225] Handling call
0831 07:59:09.884066 (+2099381us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:10.022838 (+138772us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51612928}
W20250831 07:59:10.040905 31849 leader_election.cc:336] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.093s (SENT)
W20250831 07:59:10.044507 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:10.083252 31773 connection.cc:232] RPC call timeout handler was delayed by -1.4559213689999524s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1968s for a response
I20250831 07:59:10.083436 31805 raft_consensus.cc:1238] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: []
W20250831 07:59:10.083639 31773 connection.cc:232] RPC call timeout handler was delayed by -1.4468759170000567s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19210000000000002s for a response
W20250831 07:59:10.084019 31773 connection.cc:232] RPC call timeout handler was delayed by -1.386621368999954s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20450000000000002s for a response
W20250831 07:59:10.084146 31733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1619) took 2233 ms (client timeout 2002 ms). Trace:
W20250831 07:59:10.084501 31773 connection.cc:232] RPC call timeout handler was delayed by -1.2870417170000792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1763s for a response
W20250831 07:59:10.084602 31733 rpcz_store.cc:269] 0831 07:59:07.850856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.301131 (+1450275us) service_pool.cc:225] Handling call
0831 07:59:10.083982 (+782851us) spinlock_profiling.cc:243] Waited 5.45 ms on lock 0x7b54003e2454. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:10.084074 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12534144}
I20250831 07:59:10.086117 31805 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250831 07:59:10.086400 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:10.087141 31803 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:10.088708 31804 raft_consensus.cc:1238] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 07:59:10.088883 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1550) took 2340 ms. Trace:
W20250831 07:59:10.082906 31876 rpcz_store.cc:269] 0831 07:59:06.257412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.559627 (+302215us) service_pool.cc:225] Handling call
0831 07:59:10.029030 (+3469403us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b540007dfd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:10.029254 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3273644672}
I20250831 07:59:10.089251 31881 rpcz_store.cc:276] 0831 07:59:07.748785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.766877 (+2018092us) service_pool.cc:225] Handling call
0831 07:59:09.767438 (+ 561us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:09.783551 (+ 16113us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:09.783586 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:09.793463 (+ 9877us) log.cc:844] Serialized 26 byte log entry
0831 07:59:09.999558 (+206095us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:09.999593 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:09.999655 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:10.082201 (+ 82546us) raft_consensus.cc:1683] finished
0831 07:59:10.082221 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:10.088804 (+ 6583us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":163706,"threads_started":1,"wal-append.queue_time_us":222935}
I20250831 07:59:10.090592 1297 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:10.091255 31876 raft_consensus.cc:1238] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20250831 07:59:10.084990 31773 connection.cc:232] RPC call timeout handler was delayed by -1.2509863269999641s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
W20250831 07:59:10.091800 31773 connection.cc:232] RPC call timeout handler was delayed by -1.2192747929999541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
I20250831 07:59:10.091976 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 151) took 2322 ms. Trace:
W20250831 07:59:10.092272 31773 connection.cc:232] RPC call timeout handler was delayed by -1.1217284109999355s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2766s for a response
I20250831 07:59:10.092374 31876 rpcz_store.cc:276] 0831 07:59:07.769557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.090517 (+2320960us) service_pool.cc:225] Handling call
0831 07:59:10.090977 (+ 460us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:10.091680 (+ 703us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:10.091903 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30592}
I20250831 07:59:10.093917 1334 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:10.093998 31876 raft_consensus.cc:1238] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:10.094117 1123 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:10.094858 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1556) took 2276 ms. Trace:
I20250831 07:59:10.094818 1123 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:10.048449 1190 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:10.096159 31849 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:10.053642 31731 rpcz_store.cc:269] 0831 07:59:06.235415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.239003 (+ 3588us) service_pool.cc:225] Handling call
0831 07:59:10.052776 (+3813773us) spinlock_profiling.cc:243] Waited 2.94 s on lock 0x7b540013fe54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:10.052961 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6754769280}
I20250831 07:59:10.053678 31732 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:59:10.051326 995 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:10.055668 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.040707 31878 rpcz_store.cc:276] 0831 07:59:07.542039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.274894 (+1732855us) service_pool.cc:225] Handling call
0831 07:59:09.275606 (+ 712us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:09.276612 (+ 1006us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:09.276671 (+ 59us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:09.277511 (+ 840us) log.cc:844] Serialized 26 byte log entry
0831 07:59:09.465269 (+187758us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:09.465323 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:09.465462 (+ 139us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:10.021494 (+556032us) raft_consensus.cc:1683] finished
0831 07:59:10.021524 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:10.021698 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":224853,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":354,"thread_start_us":187106,"threads_started":1,"wal-append.queue_time_us":515752}
I20250831 07:59:10.100687 1324 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:10.101564 31698 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:10.101890 31878 raft_consensus.cc:1238] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:10.102535 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1557) took 2272 ms. Trace:
I20250831 07:59:10.091254 31881 raft_consensus.cc:1238] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:10.102860 31878 rpcz_store.cc:276] 0831 07:59:07.829734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.101159 (+2271425us) service_pool.cc:225] Handling call
0831 07:59:10.101709 (+ 550us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:10.102281 (+ 572us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:10.102461 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:09.886953 1234 consensus_queue.cc:237] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:10.054888 1331 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:10.062146 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:10.066157 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:10.108523 1231 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:10.108839 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:10.109295 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:10.109505 1331 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:10.109797 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1554) took 2321 ms. Trace:
I20250831 07:59:10.110270 31881 rpcz_store.cc:276] 0831 07:59:07.786766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.090566 (+2303800us) service_pool.cc:225] Handling call
0831 07:59:10.090977 (+ 411us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:10.104723 (+ 13746us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:10.107595 (+ 2872us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36608}
W20250831 07:59:10.111019 31728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1160) took 3169 ms (client timeout 1910 ms). Trace:
I20250831 07:59:10.067147 32463 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
W20250831 07:59:10.067551 1243 consensus_meta.cc:287] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 1.755s user 0.004s sys 0.013s
I20250831 07:59:10.118183 1243 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:10.121388 32463 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38366 (request call id 402) took 2367 ms (client timeout 1591 ms). Trace:
W20250831 07:59:10.068222 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 205040us
I20250831 07:59:10.069644 1044 raft_consensus.cc:695] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:10.070567 1274 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:10.079775 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:10.080324 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:10.080729 31847 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:10.118381 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:10.086055 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:10.094626 1334 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:09.845666 1322 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:09.878636 1281 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:10.095415 31876 rpcz_store.cc:276] 0831 07:59:07.818155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.093331 (+2275176us) service_pool.cc:225] Handling call
0831 07:59:10.093704 (+ 373us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:10.094629 (+ 925us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:10.094808 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.035684 1188 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:10.097131 1123 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:10.099710 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:10.060710 32436 connection.cc:232] RPC call timeout handler was delayed by -1.6676858119999451s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18100000000000002s for a response
W20250831 07:59:10.104545 31732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 146) took 1815 ms (client timeout 1824 ms). Trace:
I20250831 07:59:10.105808 31882 raft_consensus.cc:1238] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 07:59:10.109555 31806 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:10.108861 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:10.066738 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:10.059621 1329 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:10.111500 31728 rpcz_store.cc:269] 0831 07:59:06.941342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.944211 (+ 2869us) service_pool.cc:225] Handling call
0831 07:59:10.110625 (+3166414us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b5400090454. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:10.110792 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1127090432}
I20250831 07:59:10.116564 1002 consensus_queue.cc:260] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:10.117169 1190 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:10.121546 31802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1587) took 3871 ms (client timeout 1676 ms). Trace:
I20250831 07:59:10.075420 1180 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:10.103015 1324 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:10.196007 31878 raft_consensus.cc:1273] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250831 07:59:10.221648 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 197457us
I20250831 07:59:10.191094 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:10.124641 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:10.103147 1316 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:10.213414 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1593) took 2419 ms (client timeout 2146 ms). Trace:
I20250831 07:59:10.104043 1191 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:10.313273 31804 raft_consensus.cc:1238] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:10.355793 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1551) took 2606 ms. Trace:
I20250831 07:59:10.104198 954 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250831 07:59:10.212617 31727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1614) took 2363 ms (client timeout 2093 ms). Trace:
I20250831 07:59:10.187188 31803 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:10.325837 31801 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
W20250831 07:59:10.206611 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 224693us
I20250831 07:59:10.275516 31807 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:10.358697 31882 rpcz_store.cc:276] 0831 07:59:07.749077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.028140 (+2279063us) service_pool.cc:225] Handling call
0831 07:59:10.105401 (+ 77261us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:10.355564 (+250163us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:10.355742 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.359808 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1558) took 2517 ms. Trace:
I20250831 07:59:10.360002 31809 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
W20250831 07:59:10.082630 31726 rpcz_store.cc:269] 0831 07:59:07.794596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.752154 (+1957558us) service_pool.cc:225] Handling call
0831 07:59:10.037142 (+284988us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:10.361277 31727 rpcz_store.cc:269] 0831 07:59:07.849628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.303583 (+1453955us) service_pool.cc:225] Handling call
0831 07:59:10.212389 (+908806us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b5400350cd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:10.212517 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1247155584}
W20250831 07:59:10.354347 31725 rpcz_store.cc:269] 0831 07:59:07.793708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.291639 (+1497931us) service_pool.cc:225] Handling call
0831 07:59:10.213153 (+921514us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54002eb3d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:10.213327 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":848203520}
W20250831 07:59:10.355319 31732 rpcz_store.cc:269] 0831 07:59:08.282945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.052676 (+1769731us) service_pool.cc:225] Handling call
0831 07:59:10.098620 (+ 45944us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.373507 31878 rpcz_store.cc:276] 0831 07:59:07.837541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.107387 (+2269846us) service_pool.cc:225] Handling call
0831 07:59:10.195448 (+ 88061us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:10.354854 (+159406us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:10.355068 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:10.354782 31802 rpcz_store.cc:269] 0831 07:59:06.249554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.569217 (+319663us) service_pool.cc:225] Handling call
0831 07:59:10.121102 (+3551885us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b54000ce4d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:10.121294 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1569437056}
I20250831 07:59:10.185304 1249 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:10.189754 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:10.125519 32469 raft_consensus.cc:1238] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:10.371466 31733 raft_consensus.cc:2374] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4. Re-sending same reply.
I20250831 07:59:10.356385 995 raft_consensus.cc:2747] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:10.389477 31733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 140) took 2416 ms (client timeout 2190 ms). Trace:
W20250831 07:59:10.389969 31733 rpcz_store.cc:269] 0831 07:59:07.972852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.085660 (+2112808us) service_pool.cc:225] Handling call
0831 07:59:10.389379 (+303719us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.239221 1243 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:10.197242 1318 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:10.113240 1092 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.273s
I20250831 07:59:10.345954 32462 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:10.410547 1316 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:10.411980 31808 raft_consensus.cc:1238] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: []
W20250831 07:59:10.412794 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:10.384572 31849 leader_election.cc:336] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.119s (SENT)
I20250831 07:59:10.396699 31729 raft_consensus.cc:2366] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5.
I20250831 07:59:10.416347 31849 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:10.417531 1181 raft_consensus.cc:2747] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 07:59:10.418773 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 46475.7us
W20250831 07:59:10.419282 31849 connection.cc:232] RPC call timeout handler was delayed by -1.287045626000122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22290000000000001s for a response
I20250831 07:59:10.419625 31878 raft_consensus.cc:1238] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20250831 07:59:10.393235 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.420524 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1564) took 2545 ms. Trace:
W20250831 07:59:10.420809 31733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 143) took 2413 ms (client timeout 2989 ms). Trace:
W20250831 07:59:10.421228 31733 rpcz_store.cc:269] 0831 07:59:08.007808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.390838 (+2383030us) service_pool.cc:225] Handling call
0831 07:59:10.420733 (+ 29895us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.420964 31878 rpcz_store.cc:276] 0831 07:59:07.875119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.391617 (+2516498us) service_pool.cc:225] Handling call
0831 07:59:10.419131 (+ 27514us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 07:59:10.420165 (+ 1034us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:10.420432 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.195587 1147 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:10.401358 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:10.190745 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:10.430735 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:10.403544 1339 log.cc:927] Time spent T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
W20250831 07:59:10.404029 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 323443us
I20250831 07:59:10.405261 1267 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:10.416543 31802 raft_consensus.cc:1238] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:10.193177 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:10.397199 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:10.391391 1277 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:10.430075 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.402065 31731 raft_consensus.cc:2391] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:10.430291 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:10.421988 1288 log.cc:927] Time spent T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 2.430s user 0.018s sys 0.018s
I20250831 07:59:10.376484 1317 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:10.435087 1044 consensus_queue.cc:237] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:10.234846 1303 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:10.456184 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:10.463028 31726 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:10.462729 1277 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:10.428653 1002 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:10.305894 1306 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:10.121965 32463 rpcz_store.cc:269] 0831 07:59:07.753502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:07.760416 (+ 6914us) service_pool.cc:225] Handling call
0831 07:59:09.893072 (+2132656us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e5ce 000000000035a3c9 00007fdcecee5300 00007fdcecfabec4 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:10.121290 (+228218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8291584}
I20250831 07:59:10.464432 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:10.595877 31881 raft_consensus.cc:1238] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
W20250831 07:59:10.589324 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:10.477859 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:10.635922 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:10.598867 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:10.464533 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:10.549542 32467 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:10.638597 31733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1173) took 1142 ms (client timeout 1507 ms). Trace:
I20250831 07:59:10.567834 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:10.639062 31733 rpcz_store.cc:269] 0831 07:59:09.597796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.422018 (+824222us) service_pool.cc:225] Handling call
0831 07:59:10.638487 (+216469us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.608635 31807 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:10.591413 31801 raft_consensus.cc:2391] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:59:10.479105 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 145) took 2208 ms (client timeout 2626 ms). Trace:
I20250831 07:59:10.476405 31773 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:10.640717 31725 rpcz_store.cc:269] 0831 07:59:08.270614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.377165 (+2106551us) service_pool.cc:225] Handling call
0831 07:59:10.479051 (+101886us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:10.478142 31734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1164) took 3298 ms (client timeout 3337 ms). Trace:
I20250831 07:59:10.477178 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:10.644930 31809 raft_consensus.cc:2391] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:59:10.634289 31726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1166) took 1177 ms (client timeout 1534 ms). Trace:
I20250831 07:59:10.473327 31627 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43950 (request call id 121) took 8962 ms. Trace:
W20250831 07:59:10.648347 31726 rpcz_store.cc:269] 0831 07:59:09.491939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.387664 (+895725us) service_pool.cc:225] Handling call
0831 07:59:10.634207 (+246543us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:10.468889 32436 leader_election.cc:336] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.184s (SENT)
I20250831 07:59:10.396507 32464 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:10.531111 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.468184 31800 raft_consensus.cc:1273] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
W20250831 07:59:10.652362 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:10.594118 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:10.658972 32469 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:10.659166 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:10.655047 31728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 136) took 2745 ms (client timeout 2626 ms). Trace:
W20250831 07:59:10.660233 31728 rpcz_store.cc:269] 0831 07:59:07.909170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.392590 (+2483420us) service_pool.cc:225] Handling call
0831 07:59:10.654947 (+262357us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.661224 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:10.661857 1116 consensus_meta.cc:287] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.640s user 0.008s sys 0.001s
W20250831 07:59:10.598594 1240 consensus_meta.cc:287] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.541s user 0.008s sys 0.000s
I20250831 07:59:10.655630 32436 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:10.656939 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:10.642293 31734 rpcz_store.cc:269] 0831 07:59:07.180140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:07.180321 (+ 181us) service_pool.cc:225] Handling call
0831 07:59:10.477757 (+3297436us) spinlock_profiling.cc:243] Waited 2.87 s on lock 0x7b54002ea9d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:10.477966 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6611352832}
I20250831 07:59:10.664647 31847 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:10.665060 1279 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:10.665360 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:10.468549 31731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 141) took 2489 ms (client timeout 2184 ms). Trace:
I20250831 07:59:10.665678 1180 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:59:10.661917 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:10.676335 32470 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:59:10.607998 1303 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:10.469609 1317 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:10.679361 1317 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:10.668314 31807 raft_consensus.cc:2408] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250831 07:59:10.661985 31728 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:10.657191 31800 pending_rounds.cc:85] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 1
I20250831 07:59:10.683898 31878 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:10.685096 32468 raft_consensus.cc:1811] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Handling vote request from an unknown peer a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:10.684659 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:10.682986 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:10.683946 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1171) took 1189 ms. Trace:
I20250831 07:59:10.600311 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:10.594542 1322 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:10.469283 1191 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:10.590843 32462 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38366 (request call id 405) took 1246 ms. Trace:
I20250831 07:59:10.463776 1267 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:10.469481 1334 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:10.468122 1281 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:10.691879 32468 raft_consensus.cc:2408] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250831 07:59:10.691929 31728 rpcz_store.cc:276] 0831 07:59:09.494817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.660981 (+1166164us) service_pool.cc:225] Handling call
0831 07:59:10.683867 (+ 22886us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.670554 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:10.684055 31698 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:10.690055 31800 pending_rounds.cc:107] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250831 07:59:10.697721 31727 raft_consensus.cc:2391] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:10.710297 31800 raft_consensus.cc:2887] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:10.692978 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:10.711566 31809 raft_consensus.cc:2391] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:10.673377 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.708793 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.714303 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:10.715142 31807 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:10.703238 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:10.694490 31802 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:10.597393 1243 consensus_queue.cc:1046] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.094s
I20250831 07:59:10.696533 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:10.720270 31775 leader_election.cc:336] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.910s (SENT)
W20250831 07:59:10.710659 31727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1610) took 2885 ms (client timeout 2723 ms). Trace:
I20250831 07:59:10.720548 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1174) took 1113 ms. Trace:
I20250831 07:59:10.717850 31802 raft_consensus.cc:1238] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
W20250831 07:59:10.720997 31727 rpcz_store.cc:269] 0831 07:59:07.825504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.387253 (+2561749us) service_pool.cc:225] Handling call
0831 07:59:10.710569 (+323316us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.721242 31733 rpcz_store.cc:276] 0831 07:59:09.613720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.651592 (+1037872us) service_pool.cc:225] Handling call
0831 07:59:10.720478 (+ 68886us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:10.671329 31731 rpcz_store.cc:269] 0831 07:59:07.978982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.099350 (+2120368us) service_pool.cc:225] Handling call
0831 07:59:10.468482 (+369132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.722898 31725 raft_consensus.cc:2408] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 07:59:10.672153 1116 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:10.716517 31726 raft_consensus.cc:2374] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4. Re-sending same reply.
I20250831 07:59:10.712926 31728 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
W20250831 07:59:10.724900 31726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 142) took 2742 ms (client timeout 3160 ms). Trace:
I20250831 07:59:10.725080 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1179) took 1007 ms. Trace:
W20250831 07:59:10.725418 31726 rpcz_store.cc:269] 0831 07:59:07.982446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.658063 (+2675617us) service_pool.cc:225] Handling call
0831 07:59:10.724830 (+ 66767us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.725560 31728 rpcz_store.cc:276] 0831 07:59:09.718042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.708177 (+990135us) service_pool.cc:225] Handling call
0831 07:59:10.725013 (+ 16836us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.727325 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.735980 1340 raft_consensus.cc:2747] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250831 07:59:10.738063 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 73520.3us
I20250831 07:59:10.739806 31731 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:10.740689 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1182) took 1021 ms. Trace:
I20250831 07:59:10.741113 31731 rpcz_store.cc:276] 0831 07:59:09.719057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.726967 (+1007910us) service_pool.cc:225] Handling call
0831 07:59:10.740603 (+ 13636us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.648980 31627 rpcz_store.cc:276] 0831 07:59:01.510669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.515675 (+ 5006us) service_pool.cc:225] Handling call
0831 07:59:10.473127 (+8957452us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:59:06.002085 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:59:06.119042 (+116957us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:59:06.140101 (+ 21059us) write_op.cc:435] Acquired schema lock
0831 07:59:06.158819 (+ 18718us) tablet.cc:660] Decoding operations
0831 07:59:06.227496 (+ 68677us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:59:06.227560 (+ 64us) write_op.cc:641] Partition lock acquired for write op
0831 07:59:06.227587 (+ 27us) tablet.cc:683] Acquiring locks for 6 operations
0831 07:59:06.228148 (+ 561us) tablet.cc:699] Row locks acquired
0831 07:59:06.228166 (+ 18us) write_op.cc:260] PREPARE: finished
0831 07:59:06.228313 (+ 147us) write_op.cc:270] Start()
0831 07:59:06.228422 (+ 109us) write_op.cc:276] Timestamp: P: 1756627146228270 usec, L: 0
0831 07:59:06.228444 (+ 22us) op_driver.cc:348] REPLICATION: starting
0831 07:59:06.229071 (+ 627us) log.cc:844] Serialized 2430 byte log entry
0831 07:59:07.809742 (+1580671us) op_driver.cc:464] REPLICATION: finished
0831 07:59:07.978432 (+168690us) write_op.cc:301] APPLY: starting
0831 07:59:07.980426 (+ 1994us) tablet.cc:1365] starting BulkCheckPresence
0831 07:59:08.064009 (+ 83583us) tablet.cc:1368] finished BulkCheckPresence
0831 07:59:08.064152 (+ 143us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:59:08.142854 (+ 78702us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:59:08.143131 (+ 277us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0831 07:59:08.143191 (+ 60us) write_op.cc:312] APPLY: finished
0831 07:59:08.143498 (+ 307us) log.cc:844] Serialized 53 byte log entry
0831 07:59:08.143716 (+ 218us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:08.262925 (+119209us) write_op.cc:454] Released schema lock
0831 07:59:08.275565 (+ 12640us) write_op.cc:341] FINISH: Updating metrics
0831 07:59:08.442576 (+167011us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:08.475236 (+ 32660us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":146277,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":148753,"apply.run_cpu_time_us":9308,"apply.run_wall_time_us":352324,"num_ops":6,"prepare.queue_time_us":995919,"prepare.run_cpu_time_us":114294,"prepare.run_wall_time_us":1466107,"raft.queue_time_us":216388,"raft.run_cpu_time_us":4737,"raft.run_wall_time_us":30340,"replication_time_us":1581104,"thread_start_us":2472423,"threads_started":4,"wal-append.queue_time_us":725281,"wal-append.run_cpu_time_us":37987,"wal-append.run_wall_time_us":2630644}]]}
I20250831 07:59:10.671484 1240 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:10.719743 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.749774 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:10.747632 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:10.752620 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1620) took 2901 ms (client timeout 3188 ms). Trace:
W20250831 07:59:10.759770 31725 rpcz_store.cc:269] 0831 07:59:07.851085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.666778 (+2815693us) service_pool.cc:225] Handling call
0831 07:59:10.752569 (+ 85791us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.753561 31732 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
W20250831 07:59:10.754148 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 334351us
W20250831 07:59:10.761281 31849 connection.cc:232] RPC call timeout handler was delayed by -1.1604314520000116s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2723s for a response
W20250831 07:59:10.768105 31847 connection.cc:232] RPC call timeout handler was delayed by -1.4652678180000294s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1763s for a response
I20250831 07:59:10.771436 31806 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
W20250831 07:59:10.773453 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 89461.9us
I20250831 07:59:10.774732 31698 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:10.777731 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:10.762248 31734 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:59:10.781468 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:10.785501 31734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 138) took 2875 ms (client timeout 3285 ms). Trace:
W20250831 07:59:10.786051 31734 rpcz_store.cc:269] 0831 07:59:07.909832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.696238 (+2786406us) service_pool.cc:225] Handling call
0831 07:59:10.785425 (+ 89187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.746541 1279 raft_consensus.cc:2802] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:10.789592 31698 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:10.792197 31802 raft_consensus.cc:1238] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250831 07:59:10.800266 31807 raft_consensus.cc:1238] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:10.801617 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.804909 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1175) took 1190 ms. Trace:
I20250831 07:59:10.805218 31726 rpcz_store.cc:276] 0831 07:59:09.617381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.726462 (+1109081us) service_pool.cc:225] Handling call
0831 07:59:10.804861 (+ 78399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.805748 31733 raft_consensus.cc:2391] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:10.806121 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1181) took 1087 ms. Trace:
I20250831 07:59:10.806339 31733 rpcz_store.cc:276] 0831 07:59:09.718718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.751058 (+1032340us) service_pool.cc:225] Handling call
0831 07:59:10.806078 (+ 55020us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.807157 31808 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:10.807644 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.809100 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:10.816936 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 348106us
W20250831 07:59:10.817444 32436 connection.cc:232] RPC call timeout handler was delayed by -1.6244632339999043s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3285s for a response
W20250831 07:59:10.817978 32436 connection.cc:232] RPC call timeout handler was delayed by -1.1621767399999499s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2989s for a response
W20250831 07:59:10.818361 32436 connection.cc:232] RPC call timeout handler was delayed by -1.0082767399999284s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31600000000000006s for a response
I20250831 07:59:10.702320 32462 rpcz_store.cc:276] 0831 07:59:09.304748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.598618 (+293870us) service_pool.cc:225] Handling call
0831 07:59:10.478692 (+880074us) spinlock_profiling.cc:243] Waited 572 ms on lock 0x7b54004c13d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:10.525677 (+ 46985us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1315057536}
I20250831 07:59:10.819139 31801 raft_consensus.cc:2408] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:10.821107 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.825932 31805 raft_consensus.cc:1273] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250831 07:59:10.816035 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:10.826869 31734 raft_consensus.cc:2391] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:10.792827 1303 raft_consensus.cc:3053] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:10.827621 31725 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:10.827479 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1176) took 1181 ms. Trace:
I20250831 07:59:10.833421 31734 rpcz_store.cc:276] 0831 07:59:09.687402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.786926 (+1099524us) service_pool.cc:225] Handling call
0831 07:59:10.827403 (+ 40477us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:10.833451 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1606) took 3017 ms (client timeout 3717 ms). Trace:
I20250831 07:59:10.826296 31805 pending_rounds.cc:85] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 1
W20250831 07:59:10.835058 31725 rpcz_store.cc:269] 0831 07:59:07.816465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.777396 (+2960931us) service_pool.cc:225] Handling call
0831 07:59:10.833399 (+ 56003us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.819718 31803 raft_consensus.cc:1273] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250831 07:59:10.825440 31726 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:59:10.836900 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1169) took 1343 ms. Trace:
I20250831 07:59:10.837301 31726 rpcz_store.cc:276] 0831 07:59:09.493885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.805674 (+1311789us) service_pool.cc:225] Handling call
0831 07:59:10.836825 (+ 31151us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:10.827950 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 351718us
I20250831 07:59:10.829360 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.840536 31878 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:10.810088 1279 raft_consensus.cc:695] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:10.836362 31803 pending_rounds.cc:85] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 1
I20250831 07:59:10.846977 31803 pending_rounds.cc:107] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250831 07:59:10.847294 31803 raft_consensus.cc:2887] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:10.814961 1243 raft_consensus.cc:2802] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 07:59:10.843971 31880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1172) took 4276 ms (client timeout 1517 ms). Trace:
I20250831 07:59:10.847566 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.851539 31734 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:10.852049 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1183) took 1132 ms. Trace:
W20250831 07:59:10.852099 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 142284us
I20250831 07:59:10.852049 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1177) took 1135 ms. Trace:
I20250831 07:59:10.852793 31733 rpcz_store.cc:276] 0831 07:59:09.719423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.806772 (+1087349us) service_pool.cc:225] Handling call
0831 07:59:10.851985 (+ 45213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.853025 31734 rpcz_store.cc:276] 0831 07:59:09.717207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.834305 (+1117098us) service_pool.cc:225] Handling call
0831 07:59:10.851985 (+ 17680us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:10.849485 31880 rpcz_store.cc:269] 0831 07:59:06.470042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.575101 (+105059us) service_pool.cc:225] Handling call
0831 07:59:10.709027 (+4133926us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b5400246b54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:10.746350 (+ 37323us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":287857920}
I20250831 07:59:10.854019 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:10.855849 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.830772 1316 raft_consensus.cc:2802] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:10.858650 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1180) took 1140 ms. Trace:
I20250831 07:59:10.814606 1297 consensus_queue.cc:1046] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20250831 07:59:10.860467 31773 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:10.858999 31726 rpcz_store.cc:276] 0831 07:59:09.718382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.838124 (+1119742us) service_pool.cc:225] Handling call
0831 07:59:10.858586 (+ 20462us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.837224 1303 raft_consensus.cc:738] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:10.854781 31725 raft_consensus.cc:2374] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4. Re-sending same reply.
I20250831 07:59:10.869685 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1178) took 1152 ms. Trace:
I20250831 07:59:10.862543 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:10.864976 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:10.848204 1269 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:10.849252 1243 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:10.867105 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.867415 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:10.867996 1320 raft_consensus.cc:2802] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:10.868362 31773 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:10.862533 1319 raft_consensus.cc:2802] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:10.870195 31725 rpcz_store.cc:276] 0831 07:59:09.717677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.835963 (+1118286us) service_pool.cc:225] Handling call
0831 07:59:10.869614 (+ 33651us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.873157 31734 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:10.874904 31805 pending_rounds.cc:107] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250831 07:59:10.875180 31805 raft_consensus.cc:2887] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:10.872141 31727 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
W20250831 07:59:10.877087 31727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1167) took 1384 ms (client timeout 1782 ms). Trace:
W20250831 07:59:10.877468 31727 rpcz_store.cc:269] 0831 07:59:09.492746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.721970 (+1229224us) service_pool.cc:225] Handling call
0831 07:59:10.877026 (+155056us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.880137 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:10.859036 1316 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:10.881438 32467 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:10.882138 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1170) took 1387 ms. Trace:
I20250831 07:59:10.883811 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1565) took 3008 ms. Trace:
I20250831 07:59:10.890200 31733 rpcz_store.cc:276] 0831 07:59:09.494345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.853578 (+1359233us) service_pool.cc:225] Handling call
0831 07:59:10.882065 (+ 28487us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:10.890424 31882 rpcz_store.cc:276] 0831 07:59:07.875413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.393649 (+2518236us) service_pool.cc:225] Handling call
0831 07:59:10.421683 (+ 28034us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:10.422048 (+ 365us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 07:59:10.608470 (+186422us) log.cc:844] Serialized 14 byte log entry
0831 07:59:10.881672 (+273202us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:10.882898 (+ 1226us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:10.882954 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:10.883613 (+ 659us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:10.883747 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":171599,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":1075,"thread_start_us":272729,"threads_started":1}
I20250831 07:59:10.873901 1243 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:10.895802 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:10.876040 1319 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:10.903231 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:10.885921 1320 raft_consensus.cc:695] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:10.893102 1316 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:10.911471 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:10.912314 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.873705 1269 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:10.910696 31698 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:10.894233 1306 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:10.912796 31733 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:10.903111 1261 raft_consensus.cc:2802] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 07:59:10.920485 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 52226.3us
I20250831 07:59:10.905007 1319 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:10.928572 1317 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:10.935379 31878 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:10.938370 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:10.938814 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:10.939500 31699 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 2.240s (SENT) [suppressed 60 similar messages]
I20250831 07:59:10.941025 1269 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:10.941542 1279 consensus_queue.cc:237] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:10.942340 31732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1594) took 3148 ms (client timeout 2893 ms). Trace:
W20250831 07:59:10.942903 31732 rpcz_store.cc:269] 0831 07:59:07.794010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.392786 (+2598776us) service_pool.cc:225] Handling call
0831 07:59:10.942089 (+549303us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b540011add4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:10.942248 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":261872384}
I20250831 07:59:10.944103 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:10.944386 1320 consensus_queue.cc:237] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:10.945684 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:10.947031 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:10.948005 31728 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:10.948588 31880 raft_consensus.cc:2408] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250831 07:59:10.911588 1240 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:10.957022 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 85116.1us
I20250831 07:59:10.915930 1267 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:10.947736 1299 raft_consensus.cc:2747] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:10.922376 1261 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:10.972054 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.961871 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:10.978750 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 73466.2us
I20250831 07:59:10.981300 31878 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:59:10.982214 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:10.986299 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:10.992012 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:10.995570 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:10.996575 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:10.998005 31878 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:10.999145 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:11.000082 31878 raft_consensus.cc:2391] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:11.000963 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:11.002290 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:11.006332 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:11.012950 1345 log.cc:927] Time spent T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
I20250831 07:59:11.014628 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1562) took 3140 ms. Trace:
I20250831 07:59:11.015000 31883 rpcz_store.cc:276] 0831 07:59:07.874483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.209622 (+2335139us) service_pool.cc:225] Handling call
0831 07:59:10.210273 (+ 651us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:10.210905 (+ 632us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:10.210952 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:10.211669 (+ 717us) log.cc:844] Serialized 26 byte log entry
0831 07:59:10.788788 (+577119us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:10.788835 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:10.788906 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:11.014271 (+225365us) raft_consensus.cc:1683] finished
0831 07:59:11.014298 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:11.014462 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2212,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":158,"thread_start_us":442813,"threads_started":1,"wal-append.queue_time_us":609364}
I20250831 07:59:11.016867 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:11.016191 1243 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.017619 31883 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:59:11.018517 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:11.019061 31883 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
W20250831 07:59:11.019309 1346 log.cc:927] Time spent T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
I20250831 07:59:11.019892 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:11.021229 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 150181us
I20250831 07:59:10.977653 1261 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:11.035573 31879 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250831 07:59:11.037295 31732 raft_consensus.cc:2391] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:11.038398 31878 raft_consensus.cc:2391] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
W20250831 07:59:11.039299 31877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1165) took 4571 ms (client timeout 3345 ms). Trace:
I20250831 07:59:11.021173 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:11.039770 31877 rpcz_store.cc:269] 0831 07:59:06.468284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:07.671226 (+1202942us) service_pool.cc:225] Handling call
0831 07:59:11.022426 (+3351200us) spinlock_profiling.cc:243] Waited 2.02 s on lock 0x7b54002986d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:11.039222 (+ 16796us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4648960896}
I20250831 07:59:11.040664 31731 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:11.041096 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:11.041759 31877 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:11.041972 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.042609 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:11.023571 31773 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:11.047930 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:11.043728 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:11.052433 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 92780.2us
I20250831 07:59:11.029992 32436 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:11.056810 31877 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:11.054646 31725 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:11.057204 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:11.063253 31773 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:11.062878 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:11.055116 31882 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:11.064476 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:11.068534 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:11.077466 31880 raft_consensus.cc:2391] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 7: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:11.080017 31775 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:11.084970 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1595) took 3290 ms. Trace:
I20250831 07:59:11.085583 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1549) took 3337 ms. Trace:
I20250831 07:59:11.085932 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:11.086144 31879 rpcz_store.cc:276] 0831 07:59:07.748526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.311058 (+1562532us) service_pool.cc:225] Handling call
0831 07:59:11.035240 (+1724182us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:11.036141 (+ 901us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.085324 (+ 49183us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54002986d0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:11.085491 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3963927808}
I20250831 07:59:11.088073 1264 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:11.088505 1264 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:11.088817 1264 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.085391 31725 rpcz_store.cc:276] 0831 07:59:07.794296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.064954 (+3270658us) service_pool.cc:225] Handling call
0831 07:59:11.070977 (+ 6023us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:11.080594 (+ 9617us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:11.080642 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:11.084652 (+ 4010us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:11.084715 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.084782 (+ 67us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:11.084901 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9281,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91}
I20250831 07:59:11.088828 31877 raft_consensus.cc:1238] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:11.095674 31877 raft_consensus.cc:1238] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:11.096560 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.097220 31727 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:11.080615 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:11.098800 31800 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:11.100008 31732 raft_consensus.cc:1238] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:11.100612 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1601) took 3291 ms. Trace:
I20250831 07:59:11.100890 31732 rpcz_store.cc:276] 0831 07:59:07.808777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.099528 (+3290751us) service_pool.cc:225] Handling call
0831 07:59:11.099849 (+ 321us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:11.100377 (+ 528us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.100541 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:11.102550 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1602) took 3293 ms. Trace:
I20250831 07:59:11.102839 31732 rpcz_store.cc:276] 0831 07:59:07.809085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.101625 (+3292540us) service_pool.cc:225] Handling call
0831 07:59:11.101987 (+ 362us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:11.102205 (+ 218us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:11.102232 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:11.102302 (+ 70us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:11.102340 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.102402 (+ 62us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:11.102501 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:11.106087 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:11.106698 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:11.107364 31803 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:11.107966 31732 raft_consensus.cc:1238] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 07:59:11.106784 31802 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:11.098827 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1600) took 3303 ms. Trace:
I20250831 07:59:11.110532 31878 raft_consensus.cc:1273] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20250831 07:59:11.113675 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 56814us
I20250831 07:59:11.115674 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1559) took 3242 ms. Trace:
I20250831 07:59:11.116153 31881 rpcz_store.cc:276] 0831 07:59:07.873458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.111288 (+2237830us) service_pool.cc:225] Handling call
0831 07:59:10.186975 (+ 75687us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:10.637800 (+450825us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.109899 (+472099us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b54000f7b54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:11.115604 (+ 5705us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":671338880}
W20250831 07:59:11.082075 1329 consensus_meta.cc:287] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.681s user 0.011s sys 0.000s
I20250831 07:59:11.119808 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1563) took 3245 ms. Trace:
I20250831 07:59:11.043489 31731 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855"
dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:48724
I20250831 07:59:11.113538 1243 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:11.110173 31727 rpcz_store.cc:276] 0831 07:59:07.795717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.097898 (+3302181us) service_pool.cc:225] Handling call
0831 07:59:11.098206 (+ 308us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:11.098497 (+ 291us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:11.098524 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:11.098579 (+ 55us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:11.098610 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.098661 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:11.098757 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1402,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":94}
I20250831 07:59:10.967262 1303 consensus_queue.cc:260] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.121863 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:11.116595 31734 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:11.123963 31725 raft_consensus.cc:1238] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:11.119573 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:11.125619 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1618) took 3269 ms. Trace:
I20250831 07:59:11.134193 31803 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:11.135773 31805 raft_consensus.cc:1273] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:11.135790 31732 rpcz_store.cc:276] 0831 07:59:07.850629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.103713 (+3253084us) service_pool.cc:225] Handling call
0831 07:59:11.107749 (+ 4036us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:11.108386 (+ 637us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.119731 (+ 11345us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:11.137383 32434 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65
W20250831 07:59:11.127971 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 146967us
I20250831 07:59:11.140622 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 135) took 3231 ms. Trace:
I20250831 07:59:11.140887 31734 rpcz_store.cc:276] 0831 07:59:07.908841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.123987 (+3215146us) service_pool.cc:225] Handling call
0831 07:59:11.140193 (+ 16206us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:11.140406 (+ 213us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:11.140423 (+ 17us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:11.140453 (+ 30us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:11.140474 (+ 21us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.140508 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:11.140580 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:11.138937 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:11.145530 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:11.120409 31876 rpcz_store.cc:276] 0831 07:59:07.874811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.387121 (+2512310us) service_pool.cc:225] Handling call
0831 07:59:10.459855 (+ 72734us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:10.462715 (+ 2860us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 07:59:10.873584 (+410869us) log.cc:844] Serialized 14 byte log entry
0831 07:59:11.119494 (+245910us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:11.119544 (+ 50us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:11.119595 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.119655 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:11.119759 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":127694,"prepare.queue_time_us":138972,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"thread_start_us":483470,"threads_started":2}
I20250831 07:59:11.138445 31731 raft_consensus.cc:604] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:11.152177 1275 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.155476 31775 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:11.156563 1018 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:11.156716 1032 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:11.157109 940 raft_consensus.cc:2802] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 07:59:11.158097 1352 log.cc:927] Time spent T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20250831 07:59:11.158317 1305 raft_consensus.cc:3053] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Stepping down as leader of term 5
W20250831 07:59:11.172210 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 113396us
I20250831 07:59:11.158421 1269 raft_consensus.cc:2802] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election won for term 3
W20250831 07:59:11.159183 1282 log.cc:927] Time spent T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 3.176s user 0.033s sys 0.053s
I20250831 07:59:11.121807 1329 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.168704 31732 raft_consensus.cc:1238] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:11.174046 32436 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:11.167502 31882 raft_consensus.cc:1273] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:11.127972 1297 consensus_queue.cc:1046] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20250831 07:59:11.127846 1317 consensus_queue.cc:1046] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20250831 07:59:11.166807 1018 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:11.186391 1018 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:11.187578 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1153) took 4931 ms. Trace:
I20250831 07:59:11.188000 31874 rpcz_store.cc:276] 0831 07:59:06.256490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:06.256663 (+ 173us) service_pool.cc:225] Handling call
0831 07:59:06.303441 (+ 46778us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:07.500767 (+1197326us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:07.500798 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:07.501265 (+ 467us) log.cc:844] Serialized 26 byte log entry
0831 07:59:07.847012 (+345747us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:07.847045 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:07.851395 (+ 4350us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:11.053440 (+3202045us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:11.187127 (+133687us) raft_consensus.cc:1683] finished
0831 07:59:11.187157 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:11.187392 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9910,"raft.queue_time_us":202739,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":16118,"spinlock_wait_cycles":45718784,"thread_start_us":1505466,"threads_started":2,"wal-append.queue_time_us":451857}
I20250831 07:59:11.189661 1303 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.166518 940 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:11.192122 940 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:11.178248 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48724 (request call id 7) took 3344 ms. Trace:
I20250831 07:59:11.192549 31731 rpcz_store.cc:276] 0831 07:59:07.830451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.043251 (+3212800us) service_pool.cc:225] Handling call
0831 07:59:11.171853 (+128602us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:11.193670 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.194778 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.195370 31731 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:59:11.212101 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 137) took 3270 ms. Trace:
I20250831 07:59:11.198385 31883 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:11.201401 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250831 07:59:11.213606 31800 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:11.202360 31848 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:11.176120 1320 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:11.174811 1305 raft_consensus.cc:738] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:11.218478 1305 consensus_queue.cc:260] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.219017 1329 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:11.219789 31726 raft_consensus.cc:2391] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:11.219966 1305 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:11.221783 1116 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:11.223467 1320 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:11.223927 1320 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.224488 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 147) took 3683 ms. Trace:
I20250831 07:59:11.224844 31875 rpcz_store.cc:276] 0831 07:59:07.541523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:09.258836 (+1717313us) service_pool.cc:225] Handling call
0831 07:59:10.719223 (+1460387us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:10.769347 (+ 50124us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:10.769385 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:10.808220 (+ 38835us) log.cc:844] Serialized 26 byte log entry
0831 07:59:10.948056 (+139836us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:10.963216 (+ 15160us) log.cc:844] Serialized 14 byte log entry
0831 07:59:10.963725 (+ 509us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:10.963811 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:11.092832 (+129021us) raft_consensus.cc:1683] finished
0831 07:59:11.092852 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:11.224324 (+131472us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b5400246b50. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:11.224430 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":14987,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":87,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":3357418752,"thread_start_us":83867,"threads_started":1,"wal-append.queue_time_us":199568}
I20250831 07:59:11.226559 1018 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.202634 1044 consensus_queue.cc:1046] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:59:11.205768 1277 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:11.207928 1293 consensus_queue.cc:1035] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:59:11.207903 1319 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.234467 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1168) took 1741 ms. Trace:
W20250831 07:59:11.209138 31729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1617) took 3358 ms (client timeout 2229 ms). Trace:
W20250831 07:59:11.254731 31729 rpcz_store.cc:269] 0831 07:59:07.850388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:10.012117 (+2161729us) service_pool.cc:225] Handling call
0831 07:59:11.208923 (+1196806us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b5400010c54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:11.209049 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":657967360}
I20250831 07:59:11.197520 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1598) took 3402 ms. Trace:
I20250831 07:59:11.213284 31732 rpcz_store.cc:276] 0831 07:59:07.909511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.136988 (+3227477us) service_pool.cc:225] Handling call
0831 07:59:11.168473 (+ 31485us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:11.179940 (+ 11467us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.180177 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:11.181056 1269 raft_consensus.cc:695] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 3 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:11.257583 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.179141 1243 consensus_queue.cc:1046] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
W20250831 07:59:11.216326 1353 log.cc:927] Time spent T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.279s user 0.004s sys 0.000s
I20250831 07:59:11.205739 1032 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:11.236526 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:11.257957 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 55667us
I20250831 07:59:11.231400 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250831 07:59:11.257614 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 108413us
I20250831 07:59:11.255789 31725 rpcz_store.cc:276] 0831 07:59:07.795144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.093846 (+3298702us) service_pool.cc:225] Handling call
0831 07:59:11.123768 (+ 29922us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:11.136425 (+ 12657us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.197307 (+ 60882us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b5400414450. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:11.197440 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67884288}
I20250831 07:59:11.228790 31728 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:11.228709 1018 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:11.262964 31848 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:11.257082 31726 rpcz_store.cc:276] 0831 07:59:09.493318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.233511 (+1740193us) service_pool.cc:225] Handling call
0831 07:59:11.233882 (+ 371us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:11.234120 (+ 238us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:11.234144 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:11.234195 (+ 51us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:11.234250 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.234311 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:11.234410 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:11.263429 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1623) took 3381 ms. Trace:
I20250831 07:59:11.262486 31731 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:11.279286 31775 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:11.280584 31848 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:11.281924 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1185) took 1513 ms. Trace:
I20250831 07:59:11.282406 31728 rpcz_store.cc:276] 0831 07:59:09.768335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.268998 (+1500663us) service_pool.cc:225] Handling call
0831 07:59:11.269364 (+ 366us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:11.281453 (+ 12089us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:11.281495 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:11.281557 (+ 62us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:11.281607 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.281690 (+ 83us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:11.281829 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:11.287922 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 132990us
I20250831 07:59:11.276574 31727 rpcz_store.cc:276] 0831 07:59:07.879269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.128324 (+3249055us) service_pool.cc:225] Handling call
0831 07:59:11.177281 (+ 48957us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:11.260010 (+ 82729us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:11.260048 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:11.260119 (+ 71us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:11.260163 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.260246 (+ 83us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:11.260415 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:11.293401 31732 raft_consensus.cc:1238] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:11.294212 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1172) took 1798 ms. Trace:
I20250831 07:59:11.294577 31732 rpcz_store.cc:276] 0831 07:59:09.495411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.256489 (+1761078us) service_pool.cc:225] Handling call
0831 07:59:11.275599 (+ 19110us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:11.293899 (+ 18300us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.294148 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:11.260418 1279 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:11.298272 31726 raft_consensus.cc:1238] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:11.300629 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:11.302747 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1184) took 1583 ms. Trace:
I20250831 07:59:11.303223 31729 rpcz_store.cc:276] 0831 07:59:09.719764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.259732 (+1539968us) service_pool.cc:225] Handling call
0831 07:59:11.284504 (+ 24772us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:11.302325 (+ 17821us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:11.302365 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:11.302427 (+ 62us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:11.302479 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.302554 (+ 75us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:11.302682 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:11.305100 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:11.307900 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:11.211755 940 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.310446 31731 raft_consensus.cc:1238] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:11.311965 31848 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:11.306766 1018 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:11.291500 1044 raft_consensus.cc:2762] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20250831 07:59:11.321395 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 54521.8us
I20250831 07:59:11.322125 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.310210 1240 raft_consensus.cc:3053] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Stepping down as leader of term 3
W20250831 07:59:11.324927 31849 leader_election.cc:336] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.229s (SENT)
I20250831 07:59:11.314839 1018 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:11.301661 1329 consensus_queue.cc:1046] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20250831 07:59:11.328626 31849 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:11.331987 31698 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:11.332253 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 52144.2us
I20250831 07:59:11.313479 1180 consensus_queue.cc:1046] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20250831 07:59:11.330271 1018 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.307116 1269 consensus_queue.cc:237] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.366984 1261 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.337687 1264 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.372148 1180 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:11.371937 31630 catalog_manager.cc:5582] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 2 to 4, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 4 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } }
I20250831 07:59:11.372931 1180 raft_consensus.cc:3053] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:11.373334 1180 raft_consensus.cc:738] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:11.373944 1264 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:11.374166 1180 consensus_queue.cc:260] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.364812 1253 raft_consensus.cc:2802] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:11.375142 1253 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:11.375540 1253 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:11.331521 1044 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250831 07:59:11.375932 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 44121.5us
I20250831 07:59:11.376226 1044 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:11.376657 1044 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.377259 31698 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:11.363204 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 55685.5us
I20250831 07:59:11.365482 1319 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:11.326081 1240 raft_consensus.cc:738] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:11.366073 940 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:11.381307 31775 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:11.373849 31630 catalog_manager.cc:5582] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 2 to 3, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 3 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:59:11.365417 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:59:11.379070 31848 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:11.329730 1275 consensus_queue.cc:1046] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20250831 07:59:11.402877 31848 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:11.369161 1261 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:11.356189 1329 consensus_queue.cc:1230] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20250831 07:59:11.394989 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 73670.2us
W20250831 07:59:11.417057 31849 leader_election.cc:336] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 3.188s (SENT)
I20250831 07:59:11.417652 31849 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:11.391238 1279 raft_consensus.cc:695] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:11.426581 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:11.419660 1329 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:11.423918 1361 log.cc:927] Time spent T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20250831 07:59:11.423323 31630 catalog_manager.cc:5582] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 4, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 4 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:59:11.425820 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:11.433738 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:11.429144 1329 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.441599 1260 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:11.442027 1260 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:11.442543 31804 raft_consensus.cc:1273] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 07:59:11.443727 31805 raft_consensus.cc:1273] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20250831 07:59:11.443778 31775 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:11.420085 1267 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:11.451392 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:11.539726 31730 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
W20250831 07:59:11.463900 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 86704.8us
I20250831 07:59:11.541798 31630 catalog_manager.cc:5582] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 3, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 3 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:59:11.543780 31698 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:11.544847 1116 raft_consensus.cc:2802] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:11.545264 1116 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:11.545120 31630 catalog_manager.cc:5582] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:59:11.545810 1116 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:11.483603 1367 consensus_queue.cc:1035] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.093s
I20250831 07:59:11.548121 1180 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:11.549362 1367 consensus_queue.cc:1035] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:59:11.476799 32434 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:11.551179 1269 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:59:11.551223 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 74520.5us
I20250831 07:59:11.552019 31698 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:11.491173 1240 consensus_queue.cc:260] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.553059 31698 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:11.553385 1240 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:11.554406 31698 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:11.488615 1002 raft_consensus.cc:2747] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:11.555238 31698 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:11.554983 1116 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.442401 1260 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:11.489351 1369 raft_consensus.cc:2802] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:11.556489 1369 raft_consensus.cc:695] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:11.454777 1299 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.102s
I20250831 07:59:11.483319 1317 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:11.488837 1334 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250831 07:59:11.557277 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 167570us
I20250831 07:59:11.547876 31630 catalog_manager.cc:5582] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 3, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 3 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:59:11.558333 1267 raft_consensus.cc:2802] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:11.415720 1241 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election won for term 6
W20250831 07:59:11.541253 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 145897us
I20250831 07:59:11.431833 1368 raft_consensus.cc:2802] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election won for term 4
W20250831 07:59:11.583349 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 139643us
I20250831 07:59:11.589979 31775 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:11.560082 1231 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:11.557277 1116 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:11.585695 1363 log.cc:927] Time spent T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.197s user 0.001s sys 0.000s
W20250831 07:59:11.593941 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 176930us
W20250831 07:59:11.594810 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 143397us
I20250831 07:59:11.581164 1319 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:11.585040 1334 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:11.591650 1359 log.cc:927] Time spent T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.241s user 0.000s sys 0.001s
I20250831 07:59:11.585211 1267 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:11.608764 31775 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:11.589202 1275 consensus_queue.cc:1230] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20250831 07:59:11.604589 1364 log.cc:927] Time spent T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.209s user 0.001s sys 0.000s
W20250831 07:59:11.605662 1362 log.cc:927] Time spent T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.190s user 0.000s sys 0.000s
W20250831 07:59:11.606351 1365 log.cc:927] Time spent T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.151s user 0.000s sys 0.000s
I20250831 07:59:11.592003 1261 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:11.601922 31630 catalog_manager.cc:5582] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
W20250831 07:59:11.617738 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:11.594354 1368 raft_consensus.cc:695] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:11.628296 1018 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.628933 1368 consensus_queue.cc:237] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.596638 1241 raft_consensus.cc:695] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:11.582855 1279 consensus_queue.cc:237] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:11.602536 1116 raft_consensus.cc:2802] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:11.594370 1231 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.620913 1191 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:11.627795 32436 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:11.628826 31630 catalog_manager.cc:5582] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 2 to 4, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 4 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:59:11.601958 1334 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.596014 1369 consensus_queue.cc:237] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.647496 31732 raft_consensus.cc:1238] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:11.599076 1319 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:11.631371 31803 raft_consensus.cc:1238] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:11.654295 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.661128 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:11.610214 1267 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:59:11.652530 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 54934.2us
W20250831 07:59:11.633291 31730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1607) took 3816 ms (client timeout 1846 ms). Trace:
I20250831 07:59:11.655490 31775 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:11.643699 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:11.670455 31730 rpcz_store.cc:269] 0831 07:59:07.816783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:08.301264 (+484481us) service_pool.cc:225] Handling call
0831 07:59:11.633090 (+3331826us) spinlock_profiling.cc:243] Waited 3.01 s on lock 0x7b5400091354. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:11.633215 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6932142976}
I20250831 07:59:11.670986 31726 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:11.671217 31775 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:11.669739 31630 catalog_manager.cc:5582] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 reported cstate change: term changed from 3 to 5, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 5 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:59:11.672552 31775 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:11.677793 31733 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:11.675560 31775 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:11.645013 1366 log.cc:927] Time spent T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.215s user 0.001s sys 0.001s
W20250831 07:59:11.630349 1316 consensus_meta.cc:287] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.706s user 0.009s sys 0.000s
W20250831 07:59:11.681576 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 59279.3us
I20250831 07:59:11.648478 1277 raft_consensus.cc:3053] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:11.646668 1293 raft_consensus.cc:3053] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:11.690609 1293 raft_consensus.cc:738] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:11.687788 1188 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:11.691267 1188 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:11.622051 1002 consensus_queue.cc:1035] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.058s
I20250831 07:59:11.691687 1188 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:11.691479 1044 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.646653 1243 raft_consensus.cc:3053] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:11.692622 1243 raft_consensus.cc:738] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:11.647365 1260 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.661418 1241 consensus_queue.cc:237] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.693490 1044 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:11.702575 1305 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:11.705214 1305 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.682008 1032 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:11.707459 1032 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:11.688148 1299 raft_consensus.cc:2802] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:11.707908 1032 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.708227 1299 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:11.708621 1299 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:11.630391 1018 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:11.726132 1231 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:11.687832 1277 raft_consensus.cc:738] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:11.728669 1277 consensus_queue.cc:260] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.729904 1277 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:11.731220 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:11.648356 1372 consensus_queue.cc:1046] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.084s
I20250831 07:59:11.732816 1372 raft_consensus.cc:3053] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:11.732861 1264 consensus_queue.cc:1046] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250831 07:59:11.733356 1372 raft_consensus.cc:738] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:11.733881 1264 raft_consensus.cc:3053] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:11.734234 1264 raft_consensus.cc:738] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:11.651131 1320 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.736797 1320 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:11.738318 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:11.735248 1264 consensus_queue.cc:260] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.660274 1116 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:11.742357 1116 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:11.705767 1260 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:11.652956 1329 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:11.693418 1243 consensus_queue.cc:260] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.746116 1243 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.691439 1293 consensus_queue.cc:260] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.683522 31775 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:11.748402 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 64958.4us
I20250831 07:59:11.748579 1293 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:11.749110 31726 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:11.749375 31775 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:11.686545 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 77509.6us
I20250831 07:59:11.693709 1317 consensus_queue.cc:1046] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20250831 07:59:11.752564 1317 consensus_queue.cc:1046] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:59:11.753284 1317 consensus_queue.cc:1230] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250831 07:59:11.753046 1376 consensus_queue.cc:1046] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:11.753772 1317 raft_consensus.cc:3053] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:11.754190 1317 raft_consensus.cc:738] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:59:11.696674 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 68968us
I20250831 07:59:11.754665 1303 raft_consensus.cc:3053] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:11.754959 31803 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.755153 1303 raft_consensus.cc:738] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:11.756402 1303 consensus_queue.cc:260] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.757721 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:11.758013 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:11.758531 32462 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:11.759104 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.680333 1261 raft_consensus.cc:2802] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:11.756368 1264 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.740517 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.740870 1299 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.741341 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.753816 1382 consensus_queue.cc:1035] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:59:11.757052 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:11.798137 32468 raft_consensus.cc:2408] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250831 07:59:11.648165 1275 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:11.633157 1253 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.752961 31698 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:11.701980 31729 raft_consensus.cc:2466] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:11.670145 1319 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.723119 1032 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:11.740199 31800 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:11.670364 1191 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:11.689368 1316 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:11.795387 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 44634.1us
I20250831 07:59:11.759269 1303 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.796479 31773 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:11.810382 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.796986 31725 raft_consensus.cc:2391] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:11.768460 1018 raft_consensus.cc:2802] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:11.768267 1260 raft_consensus.cc:2802] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:11.768347 1261 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:11.822149 31803 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
W20250831 07:59:11.817113 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 64235.4us
W20250831 07:59:11.822783 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:11.826442 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:11.800936 1275 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:11.852403 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:11.810081 1376 consensus_queue.cc:1230] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250831 07:59:11.869167 31848 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:11.859592 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.860433 31729 raft_consensus.cc:2374] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5. Re-sending same reply.
I20250831 07:59:11.861938 1275 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:11.862484 31801 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:11.865576 32464 raft_consensus.cc:1238] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20250831 07:59:11.883852 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:11.883852 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 148) took 3585 ms. Trace:
I20250831 07:59:11.813573 1191 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.885104 31727 raft_consensus.cc:1238] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:11.815165 1180 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:11.817135 940 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:11.956321 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:11.955734 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 141265us
I20250831 07:59:11.923578 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.958714 31725 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:11.961321 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:11.962360 31879 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:11.972277 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:11.972786 1299 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:11.972955 32462 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:11.973706 32468 raft_consensus.cc:1238] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:11.826095 1231 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:11.859316 1261 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:11.884794 31734 rpcz_store.cc:276] 0831 07:59:08.283572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.141558 (+2857986us) service_pool.cc:225] Handling call
0831 07:59:11.179057 (+ 37499us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 07:59:11.229104 (+ 50047us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:11.229134 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 07:59:11.229999 (+ 865us) log.cc:844] Serialized 52 byte log entry
0831 07:59:11.323980 (+ 93981us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:11.335754 (+ 11774us) log.cc:844] Serialized 14 byte log entry
0831 07:59:11.644315 (+308561us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:11.680387 (+ 36072us) log.cc:844] Serialized 14 byte log entry
0831 07:59:11.680635 (+ 248us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:11.680702 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:11.745217 (+ 64515us) spinlock_profiling.cc:243] Waited 2.65 ms on lock 0x7b180041c0e0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b8bf 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:59:11.868188 (+122971us) raft_consensus.cc:1683] finished
0831 07:59:11.868222 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:11.868434 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44795,"prepare.queue_time_us":55936,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":3227,"raft.queue_time_us":99723,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":8683,"spinlock_wait_cycles":57481472,"thread_start_us":385306,"threads_started":2,"wal-append.queue_time_us":181283}
I20250831 07:59:11.981985 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:11.988787 31879 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:12.007740 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1188) took 1301 ms. Trace:
I20250831 07:59:12.008328 31727 rpcz_store.cc:276] 0831 07:59:10.706507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:11.292746 (+586239us) service_pool.cc:225] Handling call
0831 07:59:11.315467 (+ 22721us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:11.943939 (+628472us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:12.007179 (+ 63240us) spinlock_profiling.cc:243] Waited 554 ms on lock 0x7b54003e1f54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:12.007643 (+ 464us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1275137792}
I20250831 07:59:12.021386 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.022337 31728 raft_consensus.cc:2391] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
W20250831 07:59:12.033360 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:11.734180 1372 consensus_queue.cc:260] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:11.953565 1180 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:11.884572 1317 consensus_queue.cc:260] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:12.266587 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:12.276937 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:12.277529 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.054440 31731 raft_consensus.cc:2408] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20250831 07:59:12.168488 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 68501.9us
W20250831 07:59:12.176260 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.301921 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.302220 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:12.032219 1231 raft_consensus.cc:695] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:12.295778 31847 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:12.277107 1334 consensus_meta.cc:287] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.603s user 0.005s sys 0.005s
I20250831 07:59:12.325970 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:12.120071 1383 log.cc:927] Time spent T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.175s user 0.001s sys 0.000s
W20250831 07:59:12.098599 1389 log.cc:927] Time spent T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.291s user 0.001s sys 0.000s
I20250831 07:59:12.135983 1372 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:12.052464 1253 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:12.087455 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:12.090497 1391 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:12.122557 1191 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.069046 1188 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.360345 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:12.369258 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.360124 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:12.361518 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.370468 32470 raft_consensus.cc:2391] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:12.360792 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.371927 31803 raft_consensus.cc:2408] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250831 07:59:12.072701 1240 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election won for term 5
W20250831 07:59:12.360559 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.286870 1367 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:12.388517 1367 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.375340 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:12.390931 1275 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.380820 32464 raft_consensus.cc:1273] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 07:59:12.273962 1393 consensus_queue.cc:1046] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.107s
I20250831 07:59:12.381356 940 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:12.401158 940 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:12.381878 1191 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:12.404794 1191 raft_consensus.cc:3053] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:12.405253 1191 raft_consensus.cc:738] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:12.406222 1191 consensus_queue.cc:260] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.384470 1188 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:12.385040 31806 raft_consensus.cc:1238] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:12.409443 1329 raft_consensus.cc:3053] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:12.409955 1329 raft_consensus.cc:738] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:12.410900 1329 consensus_queue.cc:260] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.391743 1367 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:12.431047 1329 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:12.441895 1264 consensus_meta.cc:287] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.639s user 0.005s sys 0.003s
I20250831 07:59:12.123739 1267 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.541306 1267 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:12.336918 1334 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.542735 31805 raft_consensus.cc:1238] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:12.543622 1334 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:12.416960 1269 consensus_queue.cc:1046] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250831 07:59:12.545605 1269 raft_consensus.cc:3053] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:12.546123 1269 raft_consensus.cc:738] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:12.418484 1317 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:12.345570 1032 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:12.342357 1396 consensus_queue.cc:1046] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.082s
I20250831 07:59:12.549590 1396 raft_consensus.cc:3053] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:12.550081 1396 raft_consensus.cc:738] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:12.431577 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:12.552479 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:12.338217 1116 consensus_meta.cc:287] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.595s user 0.008s sys 0.001s
I20250831 07:59:12.552987 1116 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.554981 1116 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:12.407627 32462 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:12.450377 940 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:12.378736 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 666234us
I20250831 07:59:12.556651 31773 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:12.557546 940 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:12.557883 31773 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:12.558322 940 raft_consensus.cc:2802] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:12.558720 940 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:12.559041 940 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:12.560035 1032 raft_consensus.cc:2747] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:12.559054 31773 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:12.561723 1032 raft_consensus.cc:2747] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:12.563377 31773 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:12.551513 1396 consensus_queue.cc:260] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.569134 1396 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:12.572165 1334 raft_consensus.cc:2747] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:12.547219 1269 consensus_queue.cc:260] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.573516 1269 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:12.451293 1293 consensus_meta.cc:287] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.702s user 0.004s sys 0.003s
I20250831 07:59:12.472555 1231 consensus_queue.cc:237] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.574695 31730 raft_consensus.cc:2408] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250831 07:59:12.069223 1388 consensus_queue.cc:1046] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.420s
I20250831 07:59:12.575438 1388 raft_consensus.cc:3053] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:12.575884 1388 raft_consensus.cc:738] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:12.463003 1277 consensus_meta.cc:287] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.733s user 0.008s sys 0.001s
I20250831 07:59:12.577271 32468 raft_consensus.cc:2408] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250831 07:59:12.576822 1388 consensus_queue.cc:260] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.335953 1279 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:12.579878 1044 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:12.580413 1044 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.580410 1279 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.580726 1378 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:12.582414 1044 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:12.485047 1180 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:12.343040 1316 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:12.591361 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:12.583117 1279 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:12.397483 1305 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:12.359098 1306 consensus_queue.cc:1046] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.128s
I20250831 07:59:12.596356 31734 raft_consensus.cc:2466] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:12.601612 1388 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:12.475864 1275 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:12.583779 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:12.593307 1319 consensus_meta.cc:287] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.777s user 0.011s sys 0.000s
I20250831 07:59:12.605054 1319 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.607098 1319 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:12.491750 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.496447 1260 raft_consensus.cc:695] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:12.490171 31731 raft_consensus.cc:1273] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250831 07:59:12.497033 31848 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.015s sys 0.029s
W20250831 07:59:12.608669 31848 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.719s user 0.015s sys 0.030s
I20250831 07:59:12.608420 1260 consensus_queue.cc:237] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:12.608940 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 739819us
W20250831 07:59:12.496851 1018 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.020s sys 0.077s
W20250831 07:59:12.611405 1018 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s user 0.020s sys 0.078s
W20250831 07:59:12.498274 31807 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.020s sys 0.024s
W20250831 07:59:12.611989 31807 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s user 0.021s sys 0.025s
W20250831 07:59:12.504901 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 226527us
I20250831 07:59:12.615233 31698 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:12.616032 31800 raft_consensus.cc:1273] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:12.616261 31698 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:12.617048 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.617581 31698 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:12.528373 31775 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.020s sys 0.013s
W20250831 07:59:12.618177 31775 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s user 0.020s sys 0.013s
I20250831 07:59:12.618324 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:12.618465 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 637816us
I20250831 07:59:12.618647 31733 raft_consensus.cc:1273] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:12.619032 31801 raft_consensus.cc:2391] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:12.619244 1191 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:12.525401 1244 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:12.512368 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 211602us
I20250831 07:59:12.617262 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:12.620231 31726 raft_consensus.cc:2374] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4. Re-sending same reply.
I20250831 07:59:12.620002 1244 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.621059 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:12.621803 31726 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:12.621942 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.622560 31803 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:12.622818 31726 raft_consensus.cc:1273] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250831 07:59:12.619669 1032 consensus_queue.cc:1046] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20250831 07:59:12.623173 31726 pending_rounds.cc:85] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 2
I20250831 07:59:12.623416 31726 pending_rounds.cc:107] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250831 07:59:12.519003 31877 raft_consensus.cc:1238] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:12.623708 1032 consensus_queue.cc:1046] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:59:12.624816 1032 raft_consensus.cc:3053] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Stepping down as leader of term 3
W20250831 07:59:12.529932 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 108733us
I20250831 07:59:12.625174 1032 raft_consensus.cc:738] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:12.625430 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:12.625555 31849 leader_election.cc:336] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.517s (SENT)
I20250831 07:59:12.626066 31731 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:12.626256 1407 raft_consensus.cc:3053] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:12.626874 1407 raft_consensus.cc:738] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:12.573589 31773 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:12.478495 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:12.592849 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.616473 31805 raft_consensus.cc:1273] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 07:59:12.617235 31806 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:12.517067 909 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:12.123736 1261 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.622084 1244 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:12.627692 31849 leader_election.cc:336] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.586s (SENT)
I20250831 07:59:12.627625 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.644894 31849 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:12.645126 31808 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
W20250831 07:59:12.646076 31849 leader_election.cc:336] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.593s (SENT)
I20250831 07:59:12.646234 1261 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:12.621083 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.646839 1369 raft_consensus.cc:2747] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 07:59:12.647269 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 73748.6us
I20250831 07:59:12.630638 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.648687 31809 raft_consensus.cc:2408] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250831 07:59:12.611765 1018 raft_consensus.cc:695] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:12.626070 1032 consensus_queue.cc:260] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.650127 1018 consensus_queue.cc:237] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.650758 1032 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:12.652225 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.652837 31803 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:12.485918 32466 raft_consensus.cc:2408] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250831 07:59:12.654009 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:12.646615 909 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.654575 32468 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:12.654760 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.656008 1275 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:12.656205 909 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:12.656509 1275 raft_consensus.cc:695] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:12.629912 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:12.657455 1275 consensus_queue.cc:237] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.658669 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.660445 31732 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:12.661000 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:12.661646 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:12.623624 31726 raft_consensus.cc:2887] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:12.662635 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.663120 31732 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:12.649329 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.664749 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:12.665714 32468 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:12.665830 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:12.666883 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:12.666989 31877 raft_consensus.cc:1273] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250831 07:59:12.667591 31726 raft_consensus.cc:2391] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:12.667966 1369 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:12.668484 1369 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.668562 31804 raft_consensus.cc:1238] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:12.668412 1384 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:12.670830 1384 raft_consensus.cc:2802] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:12.670897 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:12.671394 1384 raft_consensus.cc:695] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:12.671465 1369 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:12.672435 1384 consensus_queue.cc:237] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.674878 31730 raft_consensus.cc:2408] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:12.671535 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:12.677492 31731 raft_consensus.cc:2391] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:12.678233 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.536366 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:12.679076 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:12.679472 31849 leader_election.cc:336] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.624s (SENT)
I20250831 07:59:12.680063 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.679939 31849 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:12.601152 1316 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:12.681046 1369 raft_consensus.cc:2747] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:12.680866 1316 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:12.681424 31849 leader_election.cc:336] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.664s (SENT)
I20250831 07:59:12.681885 31849 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:12.676635 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:12.682653 1369 raft_consensus.cc:2747] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:12.682919 1316 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:12.582736 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 287011us
I20250831 07:59:12.627866 1407 consensus_queue.cc:260] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.685215 1407 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:12.601147 31725 raft_consensus.cc:2391] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:12.687580 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.689046 1369 consensus_queue.cc:1046] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:59:12.689682 1180 consensus_queue.cc:1046] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:59:12.691119 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:12.583448 1240 raft_consensus.cc:695] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:12.655548 31698 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:12.692898 1279 consensus_queue.cc:1035] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:59:12.691649 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.716087 31875 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:12.712424 31631 catalog_manager.cc:5582] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c reported cstate change: term changed from 1 to 2, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 2 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:59:12.718329 31773 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:12.719657 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 64166.9us
I20250831 07:59:12.719589 1240 consensus_queue.cc:237] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.721065 31698 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:12.722149 31698 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:12.716674 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.578826 940 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.724757 940 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:12.716997 32435 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:12.692524 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.736682 31731 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:12.737242 1264 raft_consensus.cc:2802] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:12.737680 1264 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:12.738030 1264 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:12.739415 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:12.582759 1303 consensus_meta.cc:287] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.758s user 0.009s sys 0.002s
I20250831 07:59:12.740310 32466 raft_consensus.cc:2391] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:12.717020 1369 raft_consensus.cc:3053] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:12.742877 1369 raft_consensus.cc:738] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:12.641965 1306 raft_consensus.cc:3053] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:12.743543 1306 raft_consensus.cc:738] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:12.744529 1369 consensus_queue.cc:260] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.745714 1369 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:12.717201 31730 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:12.723290 31725 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:12.729357 1367 consensus_queue.cc:1035] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250831 07:59:12.747344 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.749722 31806 raft_consensus.cc:3053] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:12.750064 31806 raft_consensus.cc:738] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:12.750725 31806 consensus_queue.cc:260] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.751981 1316 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:12.752529 1316 raft_consensus.cc:695] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:12.752645 1317 raft_consensus.cc:2747] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:12.740831 31728 raft_consensus.cc:2408] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:12.752888 31808 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:12.753434 1316 consensus_queue.cc:237] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.751569 31806 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:12.755309 1277 raft_consensus.cc:2802] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:12.755347 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.755925 1277 raft_consensus.cc:695] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:12.756381 31728 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:12.741894 1180 raft_consensus.cc:3053] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:12.757179 1180 raft_consensus.cc:738] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:12.756932 1277 consensus_queue.cc:237] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.757946 1180 consensus_queue.cc:260] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.759116 1180 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:12.643594 31699 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:12.693957 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:12.710762 1241 consensus_queue.cc:1046] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20250831 07:59:12.744522 1306 consensus_queue.cc:260] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:12.769457 1241 raft_consensus.cc:3053] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:12.770015 1241 raft_consensus.cc:738] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:12.770823 1241 consensus_queue.cc:260] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.768538 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:12.769502 1306 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:12.688329 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:12.774108 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.775462 31733 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:12.776000 1303 consensus_queue.cc:1046] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:59:12.742147 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.776930 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:12.777752 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.792045 31726 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:12.792151 31801 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:12.779098 1305 consensus_queue.cc:1046] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:59:12.780800 31809 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:12.781031 31725 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:12.781898 1241 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:12.783223 31800 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20250831 07:59:12.809777 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.809998 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:12.810451 31800 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:12.816040 31727 raft_consensus.cc:2408] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250831 07:59:12.740604 31848 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:12.784804 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.817584 1375 raft_consensus.cc:2802] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:12.817953 1375 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:12.818013 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.818375 1375 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:12.818612 31803 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:12.819617 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:12.819922 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 79394.1us
I20250831 07:59:12.822647 31848 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:12.823529 1279 raft_consensus.cc:2802] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:12.824028 1279 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:12.824355 1279 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:12.828383 1002 consensus_queue.cc:1035] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:59:12.836082 1367 consensus_queue.cc:1046] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:59:12.837659 1002 raft_consensus.cc:3053] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:12.838094 1002 raft_consensus.cc:738] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:12.838851 1002 consensus_queue.cc:260] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:12.840008 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.840101 31805 raft_consensus.cc:1238] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:12.840652 31728 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:12.846796 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:12.853756 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:12.854444 31879 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:12.864848 31726 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:12.799104 31806 raft_consensus.cc:1273] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250831 07:59:12.871373 31883 raft_consensus.cc:2408] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250831 07:59:12.874435 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 156192us
I20250831 07:59:12.874609 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:12.875406 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.784222 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:12.786072 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.787578 31804 raft_consensus.cc:1273] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:12.877924 31876 raft_consensus.cc:1238] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
W20250831 07:59:12.719739 31630 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59674 (request call id 215) took 11419 ms (client timeout 15000 ms). Trace:
I20250831 07:59:12.877866 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.794399 32435 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:12.879259 31729 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:59:12.879454 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 85197.3us
I20250831 07:59:12.879909 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:12.797586 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.880589 31805 raft_consensus.cc:2391] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:12.799948 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.800339 940 raft_consensus.cc:2762] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250831 07:59:12.886940 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:12.887612 31728 raft_consensus.cc:1273] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:59:12.801520 31732 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:12.802448 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.795595 31730 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:12.888695 31734 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:12.808319 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.889142 31733 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:12.881302 31801 raft_consensus.cc:2391] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:59:12.880831 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:12.682951 31849 leader_election.cc:336] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.676s (SENT)
I20250831 07:59:12.890861 31849 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:12.882575 32468 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:12.891167 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.870348 31875 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:12.884554 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.885464 31698 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:12.800951 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.808145 1369 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:12.887566 31882 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:12.894492 31725 raft_consensus.cc:1273] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20250831 07:59:12.894824 31725 pending_rounds.cc:85] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 1
I20250831 07:59:12.895021 31725 pending_rounds.cc:107] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 4.2
W20250831 07:59:12.890144 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 246656us
I20250831 07:59:12.892324 31806 raft_consensus.cc:2391] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:12.895259 31725 raft_consensus.cc:2887] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:12.893442 31874 raft_consensus.cc:1215] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Deduplicated request from leader. Original: 3.2->[4.3-4.3] Dedup: 4.3->[]
I20250831 07:59:12.896390 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:12.896644 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.897447 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.898499 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.899156 31804 raft_consensus.cc:2391] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:12.899605 31734 raft_consensus.cc:1238] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:12.899897 1305 consensus_queue.cc:1035] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:59:12.899989 1369 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:12.892953 1269 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:12.901172 1394 consensus_queue.cc:1046] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:59:12.901774 1269 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:12.901922 1369 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:12.902205 1269 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:12.896507 31874 raft_consensus.cc:1238] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20250831 07:59:12.902776 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.903749 31730 raft_consensus.cc:1238] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:12.909695 31725 raft_consensus.cc:1238] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:12.911213 31879 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:12.910478 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:12.911796 31875 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:12.912575 31775 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:12.903368 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:12.908663 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.916143 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:12.916463 31847 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:12.916754 31883 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:12.875319 1303 consensus_queue.cc:1230] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250831 07:59:12.917738 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.919493 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:12.917323 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:12.921111 31773 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:12.922545 31809 raft_consensus.cc:2366] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4.
I20250831 07:59:12.876286 1396 raft_consensus.cc:3053] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:12.878374 1319 raft_consensus.cc:2762] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250831 07:59:12.920614 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:12.926355 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 243415us
W20250831 07:59:12.927248 31849 leader_election.cc:336] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.748s (SENT)
I20250831 07:59:12.927299 31698 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:12.931381 31727 raft_consensus.cc:1238] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3. Ops: []
W20250831 07:59:12.932031 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:12.932225 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:12.932521 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.933007 31809 raft_consensus.cc:1238] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
W20250831 07:59:12.878618 31630 rpcz_store.cc:269] 0831 07:59:01.300739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:01.300958 (+ 219us) service_pool.cc:225] Handling call
0831 07:59:12.719675 (+11418717us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:59:11.806501 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:59:11.807270 (+ 769us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:59:11.807337 (+ 67us) write_op.cc:435] Acquired schema lock
0831 07:59:11.807365 (+ 28us) tablet.cc:660] Decoding operations
0831 07:59:12.133530 (+326165us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:59:12.133621 (+ 91us) write_op.cc:641] Partition lock acquired for write op
0831 07:59:12.133652 (+ 31us) tablet.cc:683] Acquiring locks for 9 operations
0831 07:59:12.134418 (+ 766us) tablet.cc:699] Row locks acquired
0831 07:59:12.134433 (+ 15us) write_op.cc:260] PREPARE: finished
0831 07:59:12.134595 (+ 162us) write_op.cc:270] Start()
0831 07:59:12.134753 (+ 158us) write_op.cc:276] Timestamp: P: 1756627152134534 usec, L: 0
0831 07:59:12.134775 (+ 22us) op_driver.cc:348] REPLICATION: starting
0831 07:59:12.135503 (+ 728us) log.cc:844] Serialized 3610 byte log entry
0831 07:59:12.527891 (+392388us) op_driver.cc:464] REPLICATION: finished
0831 07:59:12.536863 (+ 8972us) write_op.cc:301] APPLY: starting
0831 07:59:12.536964 (+ 101us) tablet.cc:1365] starting BulkCheckPresence
0831 07:59:12.537200 (+ 236us) tablet.cc:1368] finished BulkCheckPresence
0831 07:59:12.537319 (+ 119us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:59:12.538500 (+ 1181us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:59:12.538722 (+ 222us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0831 07:59:12.538777 (+ 55us) write_op.cc:312] APPLY: finished
0831 07:59:12.539083 (+ 306us) log.cc:844] Serialized 71 byte log entry
0831 07:59:12.540132 (+ 1049us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:12.540254 (+ 122us) write_op.cc:454] Released schema lock
0831 07:59:12.540300 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0831 07:59:12.584665 (+ 44365us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:12.584708 (+ 43us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":8708,"apply.run_cpu_time_us":2974,"apply.run_wall_time_us":47781,"num_ops":9,"prepare.queue_time_us":70500,"prepare.run_cpu_time_us":7233,"prepare.run_wall_time_us":690604,"raft.queue_time_us":3751,"raft.run_cpu_time_us":7444,"raft.run_wall_time_us":12895,"replication_time_us":392924,"thread_start_us":443186,"threads_started":4,"wal-append.queue_time_us":385796}]]}
I20250831 07:59:12.888741 1260 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:12.934294 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:12.934872 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:12.932526 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:12.935441 31882 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:12.935884 31732 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:12.935067 31801 raft_consensus.cc:2391] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:12.938482 31883 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:12.946540 31803 raft_consensus.cc:2391] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:12.953290 31800 raft_consensus.cc:1238] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:12.914079 1385 raft_consensus.cc:3053] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:12.956059 31802 raft_consensus.cc:1238] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:12.964557 31698 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:12.959623 31731 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:12.960502 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:12.966413 31849 leader_election.cc:336] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.762s (SENT)
I20250831 07:59:12.966986 31733 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:12.988168 1388 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:12.988967 1374 raft_consensus.cc:2802] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
W20250831 07:59:12.989931 31631 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59664 (request call id 214) took 14074 ms (client timeout 15000 ms). Trace:
I20250831 07:59:12.916193 1305 consensus_queue.cc:1035] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.074s
I20250831 07:59:12.991053 1317 raft_consensus.cc:3053] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:12.995662 1002 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:12.968164 1032 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:12.925225 1316 raft_consensus.cc:2747] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:13.001596 31802 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250831 07:59:13.002377 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 89833.5us
I20250831 07:59:13.040271 31775 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:13.004583 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 78682us
I20250831 07:59:12.941247 1044 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250831 07:59:13.007268 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.007378 31804 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:13.050868 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:13.009385 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250831 07:59:13.008172 31631 rpcz_store.cc:269] 0831 07:58:58.915987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:58:58.936361 (+ 20374us) service_pool.cc:225] Handling call
0831 07:59:12.989864 (+14053503us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:59:12.722869 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:59:12.723366 (+ 497us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:59:12.723398 (+ 32us) write_op.cc:435] Acquired schema lock
0831 07:59:12.723424 (+ 26us) tablet.cc:660] Decoding operations
0831 07:59:12.723741 (+ 317us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:59:12.723796 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0831 07:59:12.723859 (+ 63us) tablet.cc:683] Acquiring locks for 1 operations
0831 07:59:12.724038 (+ 179us) tablet.cc:699] Row locks acquired
0831 07:59:12.724054 (+ 16us) write_op.cc:260] PREPARE: finished
0831 07:59:12.724182 (+ 128us) write_op.cc:270] Start()
0831 07:59:12.724282 (+ 100us) write_op.cc:276] Timestamp: P: 1756627152724146 usec, L: 0
0831 07:59:12.724301 (+ 19us) op_driver.cc:348] REPLICATION: starting
0831 07:59:12.724830 (+ 529us) log.cc:844] Serialized 538 byte log entry
0831 07:59:12.726620 (+ 1790us) op_driver.cc:464] REPLICATION: finished
0831 07:59:12.727003 (+ 383us) write_op.cc:301] APPLY: starting
0831 07:59:12.727079 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0831 07:59:12.727171 (+ 92us) tablet.cc:1368] finished BulkCheckPresence
0831 07:59:12.727257 (+ 86us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:59:12.727501 (+ 244us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:59:12.727642 (+ 141us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0831 07:59:12.727687 (+ 45us) write_op.cc:312] APPLY: finished
0831 07:59:12.727887 (+ 200us) log.cc:844] Serialized 23 byte log entry
0831 07:59:12.728029 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:12.728105 (+ 76us) write_op.cc:454] Released schema lock
0831 07:59:12.728142 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0831 07:59:12.728418 (+ 276us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:12.728445 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":139200,"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":214,"apply.run_cpu_time_us":1375,"apply.run_wall_time_us":1385,"num_ops":1,"prepare.queue_time_us":120,"prepare.run_cpu_time_us":2183,"prepare.run_wall_time_us":2198,"replication_time_us":2218}]]}
I20250831 07:59:12.925693 1396 raft_consensus.cc:738] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:13.055835 31803 raft_consensus.cc:1273] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:12.962278 1018 raft_consensus.cc:2747] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:13.027288 31698 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:12.933338 1403 raft_consensus.cc:2747] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:12.962018 1385 raft_consensus.cc:738] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:13.020736 31627 catalog_manager.cc:5582] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 2 to 3, leader changed from 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 3 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:13.026508 1319 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:13.080404 31802 raft_consensus.cc:1273] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250831 07:59:13.058781 31775 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:13.061445 31875 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
W20250831 07:59:13.066141 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 152626us
I20250831 07:59:13.037722 1032 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:13.074241 31732 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
W20250831 07:59:13.075222 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 48082.8us
I20250831 07:59:13.088933 31698 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:13.076372 31627 catalog_manager.cc:5582] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 3 to 4, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 4 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:59:12.954500 1303 consensus_queue.cc:1035] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.102s
I20250831 07:59:13.049376 31848 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:13.082198 31775 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:13.011473 1317 raft_consensus.cc:738] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:59:13.086594 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 120205us
I20250831 07:59:13.092545 31698 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:13.093524 31698 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:13.047107 1044 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:13.094324 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:13.094489 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.037729 1275 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:13.090018 31627 catalog_manager.cc:5582] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: config changed from index 3 to 4, VOTER a0d845d01c8c4894895f66e3beae298c (127.29.217.1) evicted. New cstate: current_term: 1 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
W20250831 07:59:13.099843 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.100091 31773 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:13.045773 1374 raft_consensus.cc:695] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:12.963145 1260 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.108302 31876 raft_consensus.cc:2408] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250831 07:59:13.087955 1277 raft_consensus.cc:2802] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:13.092496 1334 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:13.088759 1319 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:13.118472 31729 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:13.038270 1388 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.120733 31627 catalog_manager.cc:5582] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 2, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:13.091780 1032 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:13.107897 1044 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:13.099169 1394 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:13.123755 1374 consensus_queue.cc:237] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:13.096958 1372 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:13.067481 1378 consensus_queue.cc:1046] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.046s
I20250831 07:59:13.119477 940 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:13.131652 31699 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:13.102819 1293 raft_consensus.cc:2802] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:13.102928 1116 raft_consensus.cc:2802] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:13.088469 1393 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:13.134034 1334 raft_consensus.cc:695] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:13.098999 1279 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.177066 1334 consensus_queue.cc:237] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:13.177785 1425 log.cc:927] Time spent T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20250831 07:59:13.109330 1367 consensus_queue.cc:1035] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
W20250831 07:59:13.179251 1422 log.cc:927] Time spent T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20250831 07:59:13.179332 1279 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:13.180140 31878 raft_consensus.cc:1238] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
W20250831 07:59:13.181211 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 49676.8us
W20250831 07:59:13.137434 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 45715.4us
I20250831 07:59:13.182430 31773 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:13.136291 32468 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20250831 07:59:13.129806 31627 catalog_manager.cc:5582] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 2 to 3, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 3 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:59:13.183205 32468 pending_rounds.cc:85] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: Aborting all ops after (but not including) 6
I20250831 07:59:13.183467 32468 pending_rounds.cc:107] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250831 07:59:13.183673 32468 raft_consensus.cc:2887] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:13.112730 1180 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:13.165354 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:13.167948 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.130492 1319 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:13.103328 1400 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20250831 07:59:13.109105 1403 consensus_queue.cc:1035] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.048s
I20250831 07:59:13.184464 31627 catalog_manager.cc:5582] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
W20250831 07:59:13.127509 1424 log.cc:927] Time spent T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.241s user 0.002s sys 0.000s
I20250831 07:59:13.112161 1275 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.198098 31773 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:13.198427 31627 catalog_manager.cc:5582] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
W20250831 07:59:13.199329 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:13.200909 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.201117 31627 catalog_manager.cc:5582] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 3, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 3 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:59:13.203596 31627 catalog_manager.cc:5582] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:59:13.205014 31627 catalog_manager.cc:5582] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 3, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 3 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:59:13.201894 1277 raft_consensus.cc:695] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:13.207118 1277 consensus_queue.cc:237] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.224025 1368 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:13.224742 1368 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:13.225309 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.227041 1368 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:13.228127 31849 leader_election.cc:336] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.830s (SENT)
W20250831 07:59:13.231808 31849 leader_election.cc:336] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.846s (SENT)
I20250831 07:59:13.232393 31849 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:13.182659 940 raft_consensus.cc:695] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:13.233296 1368 raft_consensus.cc:2747] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:13.238821 1388 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:13.243999 1372 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:13.244748 1375 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.136709 1396 consensus_queue.cc:260] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:13.140264 1317 consensus_queue.cc:260] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:13.249264 1431 log.cc:927] Time spent T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.260s user 0.003s sys 0.000s
I20250831 07:59:13.249596 1317 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:13.252250 1407 consensus_meta.cc:287] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.567s user 0.010s sys 0.000s
I20250831 07:59:13.252940 31808 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:13.246796 1375 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:13.246796 31805 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
W20250831 07:59:13.257519 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.264410 31882 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
W20250831 07:59:13.265949 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 183846us
I20250831 07:59:13.267252 31775 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:13.267886 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.191134 1293 raft_consensus.cc:695] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:59:13.275426 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 64443.6us
W20250831 07:59:13.275663 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 354833us
I20250831 07:59:13.188618 1393 raft_consensus.cc:695] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:13.271900 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 73884us
W20250831 07:59:13.273773 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.269140 31627 catalog_manager.cc:5582] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 2, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:59:13.211195 1191 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250831 07:59:13.288411 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 56633us
I20250831 07:59:13.285668 31627 catalog_manager.cc:5582] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 4, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 4 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:59:13.211746 1116 raft_consensus.cc:695] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:13.285982 32434 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:13.297786 1260 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:13.181166 1385 consensus_queue.cc:260] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.221963 1279 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20250831 07:59:13.271759 1384 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:13.287637 31773 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:13.303081 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:13.326876 31809 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:13.305989 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:13.306571 31805 raft_consensus.cc:1273] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250831 07:59:13.307964 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:13.190925 1180 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:13.310312 31849 leader_election.cc:336] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.874s (SENT)
I20250831 07:59:13.313140 31627 catalog_manager.cc:5582] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 2, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
W20250831 07:59:13.314464 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.315357 31699 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:13.283231 1261 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:13.283490 1388 raft_consensus.cc:2747] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250831 07:59:13.324501 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.361279 31803 raft_consensus.cc:2391] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:59:13.343849 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.326706 31775 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:13.316155 1191 consensus_queue.cc:1046] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20250831 07:59:13.378613 31849 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:13.394495 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 189088us
I20250831 07:59:13.363277 31802 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:13.304744 1407 consensus_queue.cc:1046] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20250831 07:59:13.394692 31627 catalog_manager.cc:5582] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 2 to 3, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 3 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
W20250831 07:59:13.323359 1427 log.cc:927] Time spent T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.198s user 0.002s sys 0.000s
I20250831 07:59:13.318768 1305 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:13.405377 31698 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:13.405836 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:13.412878 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 136143us
W20250831 07:59:13.413723 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 113690us
I20250831 07:59:13.401746 31627 catalog_manager.cc:5582] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 2, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 2 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:13.414470 31773 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:13.415477 31627 catalog_manager.cc:5582] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
W20250831 07:59:13.418308 1434 log.cc:927] Time spent T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20250831 07:59:13.418515 1447 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:13.419035 1447 raft_consensus.cc:695] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:59:13.420285 1432 log.cc:927] Time spent T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.149s user 0.000s sys 0.000s
I20250831 07:59:13.419893 1447 consensus_queue.cc:237] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:13.407672 1241 consensus_meta.cc:287] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.554s user 0.007s sys 0.002s
I20250831 07:59:13.399328 1372 raft_consensus.cc:2747] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:13.420976 1180 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:13.404542 1305 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:13.424511 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.407089 1303 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:13.460170 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 144621us
W20250831 07:59:13.464821 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 138175us
W20250831 07:59:13.456966 1264 consensus_meta.cc:287] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.718s user 0.005s sys 0.006s
W20250831 07:59:13.425172 1269 consensus_meta.cc:287] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.511s user 0.010s sys 0.000s
I20250831 07:59:13.467790 31775 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:13.400982 1403 consensus_queue.cc:1035] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20250831 07:59:13.325600 1260 consensus_queue.cc:1046] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.134s
I20250831 07:59:13.468546 31875 raft_consensus.cc:1273] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:13.421406 1395 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:13.360854 1384 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.457000 1444 raft_consensus.cc:2747] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:13.377480 940 consensus_queue.cc:237] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.460687 1261 raft_consensus.cc:695] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:13.412722 1393 consensus_queue.cc:237] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.463351 1305 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:13.475404 31880 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:13.502530 31880 raft_consensus.cc:604] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:13.477300 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:13.480793 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:13.481215 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 192177us
W20250831 07:59:13.506889 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.501729 31809 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:13.508473 31848 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:13.412637 1293 consensus_queue.cc:237] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.516017 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:13.474716 31775 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:13.531466 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:13.535450 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 50883.7us
W20250831 07:59:13.538843 31849 leader_election.cc:336] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.025s (SENT)
I20250831 07:59:13.472875 1440 raft_consensus.cc:3053] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:13.475236 1275 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:13.476377 1191 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:13.511288 1388 raft_consensus.cc:2747] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 07:59:13.548094 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 86753.3us
I20250831 07:59:13.520691 1260 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:13.521395 1444 raft_consensus.cc:2802] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:13.487594 1372 consensus_queue.cc:1035] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20250831 07:59:13.475672 1277 consensus_queue.cc:1035] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
W20250831 07:59:13.555946 31849 leader_election.cc:336] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.138s (SENT)
I20250831 07:59:13.482432 1303 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:13.475739 1269 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:13.475677 1264 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:13.541574 1449 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:13.544009 1440 raft_consensus.cc:738] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:13.556509 1191 raft_consensus.cc:695] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:13.554826 1260 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:13.557880 1334 raft_consensus.cc:2802] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:13.487542 1395 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:13.580243 1368 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:13.473969 1116 consensus_queue.cc:237] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.594673 31699 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:13.555913 1018 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250831 07:59:13.599050 31849 leader_election.cc:336] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.146s (SENT)
I20250831 07:59:13.603073 32468 raft_consensus.cc:1238] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:13.586869 1444 raft_consensus.cc:695] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:13.595405 1439 raft_consensus.cc:3053] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:13.599501 1260 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:13.618855 31729 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d"
dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:48724
I20250831 07:59:13.619622 31729 raft_consensus.cc:604] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
W20250831 07:59:13.639106 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:13.644779 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:13.679962 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 85369us
W20250831 07:59:13.683274 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 49935.3us
I20250831 07:59:13.614744 32466 raft_consensus.cc:1238] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250831 07:59:13.615474 1439 raft_consensus.cc:738] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:13.616459 1455 raft_consensus.cc:2747] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:13.592002 1261 consensus_queue.cc:237] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.605948 1368 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.679680 1440 consensus_queue.cc:260] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.733408 1440 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:13.734499 1453 log.cc:927] Time spent T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
I20250831 07:59:13.734702 1264 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:13.735152 1032 consensus_meta.cc:287] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.579s user 0.003s sys 0.004s
I20250831 07:59:13.613933 1191 consensus_queue.cc:237] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:13.726630 1454 consensus_queue.cc:1035] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.122s
W20250831 07:59:13.743871 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 146964us
W20250831 07:59:13.744520 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:13.751080 1441 log.cc:927] Time spent T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
W20250831 07:59:13.751775 1461 log.cc:927] Time spent T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20250831 07:59:13.598322 1449 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:13.753144 1439 consensus_queue.cc:260] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:13.755744 31849 leader_election.cc:336] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.191s (SENT)
I20250831 07:59:13.756569 31849 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:13.769038 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:13.735532 1032 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:13.774644 31881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:13.775372 31881 raft_consensus.cc:604] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:13.775801 31881 raft_consensus.cc:612] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:59:13.774168 1439 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
W20250831 07:59:13.779434 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:13.785262 31849 leader_election.cc:336] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.212s (SENT)
W20250831 07:59:13.787038 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.773224 1180 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250831 07:59:13.784215 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 369807us
I20250831 07:59:13.794039 31773 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:13.762472 1372 consensus_queue.cc:1035] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.024s
W20250831 07:59:13.788450 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 105538us
I20250831 07:59:13.797992 31699 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:13.799438 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:13.799063 31849 leader_election.cc:336] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.930s (SENT)
I20250831 07:59:13.803277 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:13.811486 1452 log.cc:927] Time spent T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.264s user 0.001s sys 0.000s
I20250831 07:59:13.813200 1395 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:13.815560 31849 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:13.818897 31699 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:13.789319 1384 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:13.789916 1303 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:13.854599 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.794395 1449 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:13.814574 1032 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:13.858165 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 383543us
I20250831 07:59:13.860520 31802 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:13.826642 1455 raft_consensus.cc:2747] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250831 07:59:13.864507 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 178724us
I20250831 07:59:13.871130 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:13.874677 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:13.877266 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 78249.1us
I20250831 07:59:13.875560 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:13.876276 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 57469.9us
I20250831 07:59:13.879642 31733 raft_consensus.cc:2466] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:13.884213 31699 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:13.886090 31849 leader_election.cc:336] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.982s (SENT)
I20250831 07:59:13.890960 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:13.891232 31883 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:13.891754 31883 raft_consensus.cc:604] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:13.892204 31883 raft_consensus.cc:612] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:59:13.892148 31848 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:13.892344 32435 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:13.892879 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.893765 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:13.894133 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:13.858974 1438 raft_consensus.cc:2747] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:13.894850 31729 raft_consensus.cc:1273] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:13.891664 31877 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:13.893576 1449 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:13.898483 32470 raft_consensus.cc:1273] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250831 07:59:13.900031 31699 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:13.901242 31699 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:13.904518 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:13.905156 31802 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:13.905293 31698 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:13.884284 1264 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:13.884757 1384 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:13.911691 31849 leader_election.cc:336] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.002s (SENT)
I20250831 07:59:13.772850 1444 consensus_queue.cc:237] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:13.894523 32464 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20250831 07:59:13.895452 1044 consensus_meta.cc:287] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.719s user 0.009s sys 0.002s
I20250831 07:59:13.906414 32435 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:13.907028 31699 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:13.934366 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:13.940083 31849 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:13.946252 31802 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
W20250831 07:59:13.944046 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 51978us
I20250831 07:59:13.925464 1447 raft_consensus.cc:2747] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:13.958487 32435 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:13.948613 1317 consensus_meta.cc:287] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.699s user 0.008s sys 0.004s
I20250831 07:59:13.927886 1455 raft_consensus.cc:2747] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:13.963568 31875 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
W20250831 07:59:13.950083 1319 consensus_meta.cc:287] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.749s user 0.000s sys 0.011s
I20250831 07:59:13.948149 1394 raft_consensus.cc:2802] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:13.944903 1372 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:14.007620 31875 raft_consensus.cc:604] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
W20250831 07:59:14.008489 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 96842.2us
I20250831 07:59:13.957916 1449 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 07:59:14.012569 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:14.014070 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:13.932650 1384 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:14.009486 1319 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:14.029819 1240 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.307s user 0.000s sys 0.010s
I20250831 07:59:13.930274 1264 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:13.950757 1044 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:13.974550 1303 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:14.011560 1317 raft_consensus.cc:2747] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:14.015923 1464 raft_consensus.cc:2747] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:14.014104 1395 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:14.039485 31882 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:14.591992 31882 raft_consensus.cc:604] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:14.047113 31802 raft_consensus.cc:1238] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:14.006726 1438 raft_consensus.cc:2747] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:14.148371 1446 log.cc:927] Time spent T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.192s user 0.000s sys 0.001s
I20250831 07:59:14.013640 1449 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:14.601953 1449 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:14.507036 31805 raft_consensus.cc:1238] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
W20250831 07:59:14.604063 1483 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975: Append to log took a long time: real 0.249s user 0.008s sys 0.003s
I20250831 07:59:14.187264 1306 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:14.110975 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:14.227497 31849 leader_election.cc:336] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.511s (SENT)
I20250831 07:59:14.055567 1394 raft_consensus.cc:695] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:14.161347 1384 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:14.149269 1457 log.cc:927] Time spent T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.532s user 0.002s sys 0.000s
I20250831 07:59:14.124716 1180 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:14.166385 32436 leader_election.cc:336] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.528s (SENT)
I20250831 07:59:14.632122 32436 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:14.208530 1430 log.cc:927] Time spent T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 1.149s user 0.005s sys 0.000s
I20250831 07:59:14.015342 1368 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:14.081681 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1609) took 1207 ms. Trace:
W20250831 07:59:14.162732 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 255768us
I20250831 07:59:14.633913 31876 rpcz_store.cc:276] 0831 07:59:12.867411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:12.895128 (+ 27717us) service_pool.cc:225] Handling call
0831 07:59:14.048299 (+1153171us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b54000f8f54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:14.074190 (+ 25891us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":416429312}
I20250831 07:59:14.634398 31699 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:14.093302 1480 log.cc:927] Time spent T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20250831 07:59:14.088002 1244 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:14.080873 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:14.349531 1269 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:14.518566 1305 consensus_meta.cc:287] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.999s user 0.012s sys 0.000s
W20250831 07:59:14.077548 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:14.268072 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:14.054919 1240 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.332s user 0.000s sys 0.010s
W20250831 07:59:14.510636 1482 log.cc:927] Time spent T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
W20250831 07:59:14.291378 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:14.524572 1260 consensus_meta.cc:287] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.837s user 0.011s sys 0.000s
I20250831 07:59:14.148167 1319 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:14.402623 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:14.566056 31698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.006s sys 0.006s
I20250831 07:59:14.591223 1303 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:14.600368 1264 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:14.605630 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:14.605489 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:14.632980 1044 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:14.607048 1369 raft_consensus.cc:3053] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Stepping down as leader of term 4
W20250831 07:59:14.608429 1393 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.003s sys 0.010s
W20250831 07:59:14.640903 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 474540us
W20250831 07:59:14.656239 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:14.657812 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:14.643445 31699 leader_election.cc:400] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4.
I20250831 07:59:14.646317 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
W20250831 07:59:14.645701 31698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.721s user 0.007s sys 0.006s
W20250831 07:59:14.648483 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:14.659085 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 753827us
W20250831 07:59:14.606771 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 379316us
I20250831 07:59:14.640978 1253 raft_consensus.cc:2747] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:14.636368 1180 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:14.647939 1303 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:14.643618 1368 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:14.660454 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:59:14.647696 1438 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:14.668337 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1675) took 1051 ms. Trace:
I20250831 07:59:14.668491 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1655) took 1760 ms. Trace:
I20250831 07:59:14.674387 31800 rpcz_store.cc:276] 0831 07:59:12.905467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:12.954360 (+ 48893us) service_pool.cc:225] Handling call
0831 07:59:13.010398 (+ 56038us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:13.075723 (+ 65325us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:13.075751 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:13.095803 (+ 20052us) log.cc:844] Serialized 26 byte log entry
0831 07:59:13.307036 (+211233us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:13.307075 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:13.307319 (+ 244us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:14.665737 (+1358418us) raft_consensus.cc:1683] finished
0831 07:59:14.665767 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:14.665955 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":741,"raft.queue_time_us":21818,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":7901,"thread_start_us":167118,"threads_started":1,"wal-append.queue_time_us":649711}
I20250831 07:59:14.669503 31698 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:14.660110 1393 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s user 0.004s sys 0.010s
W20250831 07:59:14.759950 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:14.760622 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:14.761091 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:14.672122 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:59:14.646389 1492 raft_consensus.cc:2747] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:14.631673 1306 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:14.700598 1490 log.cc:927] Time spent T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
I20250831 07:59:14.696003 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1664) took 1422 ms. Trace:
I20250831 07:59:14.691079 1394 consensus_queue.cc:237] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:14.699239 1180 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:14.646220 1244 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:14.702971 1449 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:14.691478 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:14.692394 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 49182.3us
I20250831 07:59:14.646958 1305 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:14.654253 1369 raft_consensus.cc:738] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:14.666543 1455 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:14.797312 31699 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:14.801158 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:14.648377 1260 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:14.699263 1396 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:14.751236 1491 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:14.753540 32468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38366 (request call id 425) took 1357 ms. Trace:
I20250831 07:59:14.754411 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1668) took 1424 ms. Trace:
I20250831 07:59:14.754805 1458 raft_consensus.cc:3053] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:14.756487 1241 raft_consensus.cc:991] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10: : Instructing follower 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:59:14.756603 1407 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:14.814616 1458 raft_consensus.cc:738] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:14.816027 1407 raft_consensus.cc:695] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:14.816686 1458 consensus_queue.cc:260] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:14.816918 1407 consensus_queue.cc:237] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:14.817659 1451 log.cc:927] Time spent T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 1.265s user 0.008s sys 0.010s
I20250831 07:59:14.817719 1458 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:14.812402 32468 rpcz_store.cc:276] 0831 07:59:13.395962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:13.396113 (+ 151us) service_pool.cc:225] Handling call
0831 07:59:13.398317 (+ 2204us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:13.603667 (+205350us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:14.753283 (+1149616us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54005ae454. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:14.753460 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":460857600}
I20250831 07:59:14.700572 1368 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:14.673246 31804 rpcz_store.cc:276] 0831 07:59:13.617194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:13.753112 (+135918us) service_pool.cc:225] Handling call
0831 07:59:13.753446 (+ 334us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:13.753645 (+ 199us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:13.753667 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:13.754166 (+ 499us) log.cc:844] Serialized 26 byte log entry
0831 07:59:13.864183 (+110017us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:13.864235 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:13.864325 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:14.668113 (+803788us) raft_consensus.cc:1683] finished
0831 07:59:14.668147 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:14.668280 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":84039,"threads_started":1,"wal-append.queue_time_us":158297}
W20250831 07:59:14.702652 1437 log.cc:927] Time spent T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 1.306s user 0.012s sys 0.006s
I20250831 07:59:14.696924 1395 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:14.704262 1385 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:14.757681 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 98135.3us
I20250831 07:59:14.758893 1334 raft_consensus.cc:695] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:14.787698 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 118428us
I20250831 07:59:14.784040 31801 rpcz_store.cc:276] 0831 07:59:13.273172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:13.273340 (+ 168us) service_pool.cc:225] Handling call
0831 07:59:13.307341 (+ 34001us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:13.314949 (+ 7608us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:13.314969 (+ 20us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:13.325929 (+ 10960us) log.cc:844] Serialized 26 byte log entry
0831 07:59:14.023051 (+697122us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b18004d0140. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce7c4955c 00007fdce507e77b 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdce7c3b2bb 00007fdce7c40dc3 00007fdce7c41171
0831 07:59:14.040263 (+ 17212us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:14.040325 (+ 62us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:14.040411 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:14.085094 (+ 44683us) spinlock_profiling.cc:243] Waited 4.45 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:14.695738 (+610644us) raft_consensus.cc:1683] finished
0831 07:59:14.695767 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:14.695938 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7198,"raft.queue_time_us":116693,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":6370,"spinlock_wait_cycles":15479936,"thread_start_us":70048,"threads_started":1,"wal-append.queue_time_us":187364}
I20250831 07:59:14.800819 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1667) took 1501 ms. Trace:
I20250831 07:59:14.654645 1264 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:14.804898 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 144859us
I20250831 07:59:14.812446 31803 rpcz_store.cc:276] 0831 07:59:13.330074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:13.398917 (+ 68843us) service_pool.cc:225] Handling call
0831 07:59:13.421866 (+ 22949us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 07:59:13.463992 (+ 42126us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:13.464023 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 07:59:13.471198 (+ 7175us) log.cc:844] Serialized 52 byte log entry
0831 07:59:13.510360 (+ 39162us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:13.511022 (+ 662us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:13.511182 (+ 160us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:14.693371 (+1182189us) spinlock_profiling.cc:243] Waited 4.28 ms on lock 0x7b18004d0080. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce501365a 00007fdcee4e4409 00007fdcebebce6f 00007fdce509b90e 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d
0831 07:59:14.750049 (+ 56678us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b18004d0080. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b792 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:59:14.754194 (+ 4145us) raft_consensus.cc:1683] finished
0831 07:59:14.754218 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:14.754355 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45328,"raft.queue_time_us":109592,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":10103,"spinlock_wait_cycles":51606272,"thread_start_us":22298,"threads_started":1,"wal-append.queue_time_us":83711}
I20250831 07:59:14.800181 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47886 (request call id 384) took 1918 ms. Trace:
I20250831 07:59:14.794756 1449 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:14.800127 1475 raft_consensus.cc:2747] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:14.760303 1438 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:14.845161 31802 raft_consensus.cc:1238] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:14.795097 1244 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:14.845088 32436 leader_election.cc:336] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.729s (SENT)
I20250831 07:59:14.792232 1498 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:14.858144 1241 raft_consensus.cc:1079] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Signalling peer 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:59:14.819700 1491 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:14.864328 1369 consensus_queue.cc:260] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:14.866364 1485 log.cc:927] Time spent T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.857s user 0.023s sys 0.007s
I20250831 07:59:14.808331 1492 raft_consensus.cc:2747] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:14.868325 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1603) took 2019 ms. Trace:
W20250831 07:59:14.870057 1489 log.cc:927] Time spent T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.782s user 0.000s sys 0.013s
I20250831 07:59:14.871088 1260 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:14.871454 32462 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47886 (request call id 389) took 1285 ms. Trace:
I20250831 07:59:14.809113 1455 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:14.873118 1306 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:14.872089 1305 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:15.042007 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:15.035284 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:15.040542 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 1.2465e+06us
I20250831 07:59:15.043535 31878 rpcz_store.cc:276] 0831 07:59:12.848925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:12.849047 (+ 122us) service_pool.cc:225] Handling call
0831 07:59:12.849390 (+ 343us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:13.180575 (+331185us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:14.867995 (+1687420us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b54003ef654. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:14.868239 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":760230784}
I20250831 07:59:15.044008 32436 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:15.043035 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 229213us
I20250831 07:59:15.046198 31698 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:14.890048 31808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1495) took 2060 ms (client timeout 2162 ms). Trace:
W20250831 07:59:15.046823 31808 rpcz_store.cc:269] 0831 07:59:12.784721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:12.896423 (+111702us) service_pool.cc:225] Handling call
0831 07:59:14.821126 (+1924703us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b5400084dd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:14.845006 (+ 23880us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":817944960}
I20250831 07:59:15.047313 31699 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:15.045661 31805 rpcz_store.cc:276] 0831 07:59:13.299797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:13.370755 (+ 70958us) service_pool.cc:225] Handling call
0831 07:59:13.416137 (+ 45382us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:14.604220 (+1188083us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:14.800496 (+196276us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54002580d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:14.800731 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2508164480}
W20250831 07:59:15.050294 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.050848 31731 raft_consensus.cc:1273] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:59:15.043356 32462 rpcz_store.cc:276] 0831 07:59:13.585933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:13.591097 (+ 5164us) service_pool.cc:225] Handling call
0831 07:59:13.680067 (+ 88970us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:13.794201 (+114134us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:13.795599 (+ 1398us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:13.891864 (+ 96265us) log.cc:844] Serialized 26 byte log entry
0831 07:59:13.974778 (+ 82914us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:14.006263 (+ 31485us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:14.870924 (+864661us) spinlock_profiling.cc:243] Waited 74.1 ms on lock 0x7b54003ba954. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:14.870993 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:14.871061 (+ 68us) raft_consensus.cc:1683] finished
0831 07:59:14.871083 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:14.871364 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8830,"raft.queue_time_us":8793,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":4383,"spinlock_wait_cycles":170463616,"thread_start_us":61846,"threads_started":1,"wal-append.queue_time_us":226564}
I20250831 07:59:15.048892 1447 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:15.053499 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1679) took 1093 ms. Trace:
I20250831 07:59:15.060101 31800 rpcz_store.cc:276] 0831 07:59:13.960318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:14.847075 (+886757us) service_pool.cc:225] Handling call
0831 07:59:14.868053 (+ 20978us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:14.868243 (+ 190us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:14.868270 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:14.868317 (+ 47us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:14.868353 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:14.868411 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:15.053427 (+185016us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:15.055807 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:15.053606 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.063561 31878 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:15.064069 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 219) took 2140 ms. Trace:
I20250831 07:59:15.064952 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1641) took 2442 ms. Trace:
W20250831 07:59:15.159394 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 60515.8us
I20250831 07:59:15.078915 31882 raft_consensus.cc:1273] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:59:15.081974 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1674) took 1525 ms. Trace:
I20250831 07:59:15.160243 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:15.160897 31881 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:15.077071 31732 raft_consensus.cc:1273] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20250831 07:59:15.160521 31849 leader_election.cc:336] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.723s (SENT)
I20250831 07:59:15.107910 31876 raft_consensus.cc:1273] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:59:14.896956 1395 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:15.077016 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:15.184352 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 121015us
I20250831 07:59:15.162708 31849 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:15.185359 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:15.099499 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:15.083765 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.160516 31809 rpcz_store.cc:276] 0831 07:59:13.556831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:13.558911 (+ 2080us) service_pool.cc:225] Handling call
0831 07:59:13.559587 (+ 676us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:13.588743 (+ 29156us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:13.588792 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:13.598505 (+ 9713us) log.cc:844] Serialized 26 byte log entry
0831 07:59:13.867434 (+268929us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:13.867486 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:13.867574 (+ 88us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:14.865974 (+998400us) spinlock_profiling.cc:243] Waited 8.44 ms on lock 0x7b2400279218. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4ef2e94 00007fdce4ef0f9b 00007fdce4f7fd79 00007fdce4f7fa05 00007fdce4f7f04d 00007fdce7c2520e 00007fdce7c3c93c 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1
0831 07:59:15.081680 (+215706us) raft_consensus.cc:1683] finished
0831 07:59:15.081704 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:15.081914 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53203,"raft.queue_time_us":1197297,"raft.run_cpu_time_us":495,"raft.run_wall_time_us":49441,"spinlock_wait_cycles":19422592,"thread_start_us":182686,"threads_started":1,"wal-append.queue_time_us":342884}
W20250831 07:59:15.158635 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.023792 1269 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:14.929230 1493 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:15.182679 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 137173us
I20250831 07:59:15.159579 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39068 (request call id 582) took 1597 ms. Trace:
I20250831 07:59:15.192181 31801 rpcz_store.cc:276] 0831 07:59:13.562480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:15.055761 (+1493281us) service_pool.cc:225] Handling call
0831 07:59:15.159516 (+103755us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:15.184257 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:14.979202 32464 raft_consensus.cc:2391] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:15.056264 1264 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:15.081619 1180 raft_consensus.cc:991] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10: : Instructing follower 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:59:15.065230 1400 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:15.064661 31806 rpcz_store.cc:276] 0831 07:59:12.924166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:13.029481 (+105315us) service_pool.cc:225] Handling call
0831 07:59:13.076768 (+ 47287us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:13.077198 (+ 430us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:13.077240 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:13.077806 (+ 566us) log.cc:844] Serialized 26 byte log entry
0831 07:59:13.192892 (+115086us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:13.192949 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:13.193120 (+ 171us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:14.702161 (+1509041us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:15.013633 (+311472us) raft_consensus.cc:1683] finished
0831 07:59:15.013657 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:15.064004 (+ 50347us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2449,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":7359,"spinlock_wait_cycles":12138240,"thread_start_us":114731,"threads_started":1,"wal-append.queue_time_us":238255}
I20250831 07:59:15.188663 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:15.189638 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 80467.5us
W20250831 07:59:15.189869 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 344813us
W20250831 07:59:15.108233 1502 log.cc:927] Time spent T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
W20250831 07:59:15.219856 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:15.220247 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.059518 1374 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.142s
I20250831 07:59:15.108346 1369 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:14.879312 1303 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:15.054224 1018 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:15.107303 1277 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:15.210815 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:15.257803 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 97306.6us
I20250831 07:59:15.107596 1261 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:15.212637 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:15.109887 1319 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:15.082788 940 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:15.107580 1447 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:15.212875 31773 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:15.218698 31883 raft_consensus.cc:2391] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:15.083097 1316 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:15.159657 31807 rpcz_store.cc:276] 0831 07:59:12.613529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:12.623995 (+ 10466us) service_pool.cc:225] Handling call
0831 07:59:12.624278 (+ 283us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:12.624473 (+ 195us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 07:59:12.941027 (+316554us) log.cc:844] Serialized 14 byte log entry
0831 07:59:14.877699 (+1936672us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b18004d0680. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce7c4955c 00007fdce507e77b 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdce7c3b2bb 00007fdce7c40dc3 00007fdce7c412da
0831 07:59:15.055132 (+177433us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:15.055184 (+ 52us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:15.055225 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:15.055328 (+ 103us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:15.055445 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":177168,"prepare.queue_time_us":476455,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":184544,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":42715,"spinlock_wait_cycles":36862208,"thread_start_us":490110,"threads_started":2,"wal-append.queue_time_us":322930,"wal-append.run_cpu_time_us":1818,"wal-append.run_wall_time_us":1154490}
I20250831 07:59:15.054032 1454 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.137s
I20250831 07:59:15.107888 1478 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:15.183164 1445 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:15.059971 1231 consensus_queue.cc:1046] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.150s
I20250831 07:59:15.108193 1401 raft_consensus.cc:2762] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250831 07:59:15.250129 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:15.283411 31809 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:15.250494 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:15.251323 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:15.212565 1493 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:15.162650 1305 consensus_queue.cc:1046] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
W20250831 07:59:15.259166 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:15.286978 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.222012 1375 raft_consensus.cc:2747] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:15.268280 1368 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:15.051277 32466 rpcz_store.cc:276] 0831 07:59:12.881990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:12.887291 (+ 5301us) service_pool.cc:225] Handling call
0831 07:59:12.899899 (+ 12608us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:13.713755 (+813856us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:14.786978 (+1073223us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b540068b354. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:14.800103 (+ 13125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1609103744}
I20250831 07:59:15.312749 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:15.313731 31806 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:15.314939 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:15.265304 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:15.317132 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.248445 1378 raft_consensus.cc:1079] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Signalling peer 7512ba2ed3a94982a22a75f51a647e65 to start an election
I20250831 07:59:15.266227 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:15.209645 1269 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:15.284776 31801 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:15.293394 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 80630.8us
I20250831 07:59:15.324586 31773 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:15.263787 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:15.254877 31878 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:15.278812 1477 raft_consensus.cc:2747] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:15.319144 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1678) took 1361 ms. Trace:
I20250831 07:59:15.320770 31805 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:15.260167 1334 consensus_queue.cc:237] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:15.286896 1306 consensus_queue.cc:1046] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20250831 07:59:15.261096 1018 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:15.266348 1261 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:15.331728 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:15.245818 1400 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:15.271549 1319 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:15.333032 31699 connection.cc:232] RPC call timeout handler was delayed by -3.0063693989999365s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20250831 07:59:15.337221 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:15.320930 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 84803.8us
I20250831 07:59:15.325560 31800 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
W20250831 07:59:15.354292 31699 connection.cc:232] RPC call timeout handler was delayed by -2.5635096610000119s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2162s for a response
I20250831 07:59:15.278250 1277 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:15.357216 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.341179 31809 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:15.276968 940 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:15.279976 1447 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:15.322824 1374 consensus_queue.cc:1046] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20250831 07:59:15.346171 31804 rpcz_store.cc:276] 0831 07:59:13.958075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:14.840714 (+882639us) service_pool.cc:225] Handling call
0831 07:59:14.880771 (+ 40057us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:14.882785 (+ 2014us) raft_consensus.cc:1532] Early marking committed up to index 5
0831 07:59:15.216521 (+333736us) log.cc:844] Serialized 14 byte log entry
0831 07:59:15.318788 (+102267us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:15.318866 (+ 78us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:15.318930 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:15.318989 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:15.319094 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":438475,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":2576,"raft.queue_time_us":2963,"spinlock_wait_cycles":14541184,"thread_start_us":371563,"threads_started":2}
I20250831 07:59:15.336870 1407 raft_consensus.cc:2747] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 07:59:15.379201 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 148652us
I20250831 07:59:15.359664 32435 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:15.285668 1478 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:15.375623 32436 leader_election.cc:336] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.559s (SENT)
I20250831 07:59:15.386336 32436 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:15.391501 31805 raft_consensus.cc:2408] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250831 07:59:15.393071 32435 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:15.401216 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:15.402659 31729 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:15.403849 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.405591 31806 raft_consensus.cc:1273] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:15.366537 1508 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:15.412137 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:15.412869 1512 log.cc:927] Time spent T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20250831 07:59:15.413123 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:15.413807 1511 log.cc:927] Time spent T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20250831 07:59:15.415086 1277 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:15.285559 1445 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:15.421321 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.437623 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.409734 31804 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:39082
I20250831 07:59:15.412892 31732 raft_consensus.cc:1238] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:15.438407 31804 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:59:15.439272 31804 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:15.421669 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.440785 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.438145 31729 raft_consensus.cc:2374] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5. Re-sending same reply.
I20250831 07:59:15.442080 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.417716 31731 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:15.442720 31729 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:15.422354 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.422000 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.446660 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:15.446962 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 95300.4us
I20250831 07:59:15.447295 31725 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:15.447686 31848 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:15.448294 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.451288 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.452862 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.454236 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.394208 1463 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:15.445299 31734 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:59:15.457186 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.461827 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.464394 31734 raft_consensus.cc:2408] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:15.464277 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.408805 1455 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:15.465090 31732 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:15.466266 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.467216 31732 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:15.459785 31729 raft_consensus.cc:2391] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:15.460793 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.465611 31880 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:15.476024 1463 raft_consensus.cc:695] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:15.462329 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.477093 1463 consensus_queue.cc:237] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:15.478202 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.477456 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.481499 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.481722 31730 raft_consensus.cc:2374] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5. Re-sending same reply.
I20250831 07:59:15.482223 31734 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:15.482690 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.438541 1264 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:15.415768 940 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:15.493950 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.494887 31733 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:15.484793 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.438313 1491 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:15.483335 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.500541 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.501587 31734 raft_consensus.cc:2408] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:15.502518 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.484081 31730 raft_consensus.cc:2408] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:15.504261 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.498685 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.505620 31727 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:15.507550 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.508311 31732 raft_consensus.cc:2374] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5. Re-sending same reply.
W20250831 07:59:15.509781 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.510952 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.511968 31727 raft_consensus.cc:2408] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:15.512002 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.515514 31729 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:15.513173 32468 raft_consensus.cc:1238] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 5. Ops: []
I20250831 07:59:15.463958 1303 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:15.523694 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.525524 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.520359 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.493985 1275 raft_consensus.cc:3053] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:15.463609 32462 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:15.528333 31729 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 6
I20250831 07:59:15.528512 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.540591 31733 raft_consensus.cc:2408] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250831 07:59:15.529299 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.530032 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.531339 31725 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:15.498845 1458 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:15.508252 1400 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:15.539438 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.554157 31732 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:15.555298 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:15.553929 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:15.555889 31726 raft_consensus.cc:2374] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4. Re-sending same reply.
W20250831 07:59:15.556177 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 155084us
I20250831 07:59:15.481984 1374 consensus_queue.cc:1035] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
W20250831 07:59:15.560016 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 193125us
W20250831 07:59:15.547297 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 158607us
I20250831 07:59:15.546375 31725 raft_consensus.cc:1273] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20250831 07:59:15.566164 31725 pending_rounds.cc:85] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 3
I20250831 07:59:15.566682 31725 pending_rounds.cc:107] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250831 07:59:15.567013 31725 raft_consensus.cc:2887] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:15.532119 1275 raft_consensus.cc:738] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:15.540464 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:15.558859 31804 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:15.573927 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:15.566985 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:15.556772 32436 leader_election.cc:336] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.673s (SENT)
I20250831 07:59:15.572448 31698 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:15.580682 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.578519 32468 raft_consensus.cc:1238] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:15.578189 31733 raft_consensus.cc:1273] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250831 07:59:15.581117 31733 pending_rounds.cc:85] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 1
I20250831 07:59:15.581557 31733 pending_rounds.cc:107] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250831 07:59:15.581121 32436 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:15.581863 31733 raft_consensus.cc:2887] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250831 07:59:15.582597 32436 leader_election.cc:336] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.676s (SENT)
I20250831 07:59:15.582667 31804 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:15.584012 31732 raft_consensus.cc:1273] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:15.584251 32436 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:15.585938 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:15.587806 32436 leader_election.cc:336] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.672s (SENT)
I20250831 07:59:15.589780 32468 raft_consensus.cc:1238] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 3 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:15.588332 32436 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:15.555505 1493 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:15.592659 31698 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:15.595885 31729 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
W20250831 07:59:15.594498 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 146891us
I20250831 07:59:15.600131 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:15.600196 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:15.573112 1316 raft_consensus.cc:695] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:15.601068 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:15.613548 32470 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
W20250831 07:59:15.615077 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.581739 1478 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:15.590413 1018 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:15.588989 1244 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250831 07:59:15.601011 1504 log.cc:927] Time spent T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.219s user 0.000s sys 0.001s
I20250831 07:59:15.589156 1508 raft_consensus.cc:2747] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:15.556738 1458 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:15.641863 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:15.600867 1493 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:15.604017 1319 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:15.613953 1403 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:15.598487 1264 raft_consensus.cc:2747] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:15.590548 32466 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:15.616729 1466 raft_consensus.cc:2747] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:15.596879 1472 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:15.598690 1521 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:15.582086 1400 consensus_queue.cc:1046] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20250831 07:59:15.593981 1525 consensus_queue.cc:1035] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.159s
I20250831 07:59:15.639995 1305 consensus_queue.cc:1046] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250831 07:59:15.617223 1527 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:15.623968 1447 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:15.622042 1261 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:15.660720 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:15.671289 1458 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:15.673933 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 58377.4us
I20250831 07:59:15.675189 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:15.660605 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:15.680809 31804 raft_consensus.cc:1238] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:15.660219 32463 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:15.666962 1403 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:15.716184 1403 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:15.711573 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:15.719686 31882 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:15.719910 1466 consensus_queue.cc:1046] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250831 07:59:15.691432 1527 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:15.682654 1394 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:15.670858 1493 raft_consensus.cc:695] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 6 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:15.685815 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:15.626502 1275 consensus_queue.cc:260] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:15.688232 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:15.691125 32467 raft_consensus.cc:1238] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:15.691891 32436 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
W20250831 07:59:15.861151 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 169416us
I20250831 07:59:15.862093 32436 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:15.677567 1521 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:15.682945 1526 log.cc:927] Time spent T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.170s user 0.000s sys 0.001s
I20250831 07:59:15.709920 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:15.710945 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:15.674542 1316 consensus_queue.cc:237] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:15.715571 1498 consensus_meta.cc:287] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.622s user 0.009s sys 0.000s
I20250831 07:59:15.723575 32470 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
W20250831 07:59:15.724984 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:15.874065 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:15.874598 32436 leader_election.cc:336] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.776s (SENT)
I20250831 07:59:15.664408 1538 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:15.723671 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 224) took 1949 ms. Trace:
I20250831 07:59:15.880237 31802 rpcz_store.cc:276] 0831 07:59:13.773931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:14.597504 (+823573us) service_pool.cc:225] Handling call
0831 07:59:14.600039 (+ 2535us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:15.050424 (+450385us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:15.723460 (+673036us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54000e97d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:15.723615 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":532944768}
W20250831 07:59:15.880664 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.727049 32471 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
W20250831 07:59:15.881199 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.733198 31882 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8.
I20250831 07:59:15.737181 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:15.882422 31879 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:15.823774 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:15.781642 1525 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:15.829980 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.778930 1394 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:15.787566 31801 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:15.888753 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:15.709903 1385 consensus_meta.cc:287] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.656s user 0.008s sys 0.001s
I20250831 07:59:15.690552 1472 raft_consensus.cc:3053] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:15.696390 1445 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:15.872216 1521 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:15.875176 32436 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:15.878172 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:15.884474 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:15.885707 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:15.781664 31876 raft_consensus.cc:1238] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:15.890770 32435 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:15.879586 1498 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:15.905411 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:15.910771 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 407740us
I20250831 07:59:15.871940 1548 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:15.912935 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:15.888224 940 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:15.912299 31847 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:15.942454 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:15.892890 1472 raft_consensus.cc:738] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:15.905273 1487 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:15.944774 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 54093.6us
I20250831 07:59:15.883525 1527 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:15.913758 1501 raft_consensus.cc:2747] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:15.953245 32436 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:15.892252 1525 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:15.954226 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:15.960441 31728 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:15.961544 32471 raft_consensus.cc:2391] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:15.905689 1449 consensus_queue.cc:1046] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
W20250831 07:59:15.950078 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 344259us
I20250831 07:59:15.951408 1548 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:15.892753 1538 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:15.949988 1306 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:15.971937 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:15.972007 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.973793 31727 raft_consensus.cc:1238] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250831 07:59:15.977079 31698 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:15.979183 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:15.915143 1493 consensus_queue.cc:237] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:15.984977 31698 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:15.987082 31732 raft_consensus.cc:1238] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:15.985800 32436 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:15.969473 1244 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:15.991669 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:15.965734 1385 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:15.993429 31698 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:15.994894 31733 raft_consensus.cc:2374] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6. Re-sending same reply.
I20250831 07:59:15.968113 1548 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:16.005973 1557 consensus_queue.cc:1035] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:59:16.008036 1231 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:16.008572 1231 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:16.008991 1231 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:16.012804 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.962554 940 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.015125 940 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:16.015424 1538 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:16.016018 1394 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:16.003856 1449 consensus_queue.cc:1046] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
W20250831 07:59:16.016956 1395 consensus_meta.cc:287] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.808s user 0.007s sys 0.005s
I20250831 07:59:16.017366 1395 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.019356 1395 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:16.019917 31698 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:16.020226 32470 raft_consensus.cc:1238] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 6. Current term is 7. Ops: []
W20250831 07:59:16.021045 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.021340 1531 raft_consensus.cc:2747] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 07:59:16.022239 1549 log.cc:927] Time spent T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20250831 07:59:16.028379 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:16.029135 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:15.984717 1367 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:16.035404 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:15.985288 1306 raft_consensus.cc:695] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:15.999339 1523 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:15.997222 1527 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:15.996609 1463 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:16.000852 1244 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:16.058105 1244 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:16.063848 940 consensus_queue.cc:1046] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250831 07:59:16.043736 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.000758 1385 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:16.054296 1116 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:15.962357 1487 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.064574 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.064764 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:16.039750 1557 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.042171 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:16.065130 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:16.065855 31874 raft_consensus.cc:1273] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250831 07:59:16.023197 1403 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.066210 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:16.023204 1531 consensus_queue.cc:1046] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20250831 07:59:16.094962 31807 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:16.069284 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:16.096278 31879 raft_consensus.cc:2408] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:16.074250 31773 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:16.077306 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.079609 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:16.098915 31876 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:16.100068 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:16.100780 31876 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:16.064249 1505 raft_consensus.cc:3053] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:16.078981 1385 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:16.106600 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:16.069836 1253 raft_consensus.cc:3053] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:16.089303 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:16.107048 31773 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:16.120463 31802 raft_consensus.cc:2391] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:16.068445 32466 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:16.122076 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:16.122860 32466 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:16.123971 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:16.124940 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.107405 31801 raft_consensus.cc:2466] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:16.126049 1508 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:16.108126 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:16.127769 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.128801 31773 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:16.129971 940 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election won for term 5
W20250831 07:59:16.118090 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 120954us
I20250831 07:59:16.119168 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:16.138008 31874 raft_consensus.cc:2391] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:16.138621 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.140290 31729 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:16.142515 31876 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:16.046640 1367 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.144646 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:16.150616 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.139539 31698 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:16.154251 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:16.161214 1551 log.cc:927] Time spent T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
I20250831 07:59:16.162009 1558 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.162614 1558 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.152630 1305 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
W20250831 07:59:16.143030 1485 log.cc:927] Time spent T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20250831 07:59:16.165043 1306 consensus_queue.cc:237] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.166107 1403 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:16.000460 1472 consensus_queue.cc:260] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.119918 1253 raft_consensus.cc:738] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:16.088387 1116 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.167960 1472 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:16.168342 1253 consensus_queue.cc:260] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.102483 1557 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.170621 31875 raft_consensus.cc:1273] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:16.103171 1547 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:16.103427 1505 raft_consensus.cc:738] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:16.138531 1439 raft_consensus.cc:2747] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:16.179705 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.158977 1531 consensus_queue.cc:1230] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 07:59:16.183415 31804 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:16.143481 1523 raft_consensus.cc:695] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 8 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:16.160079 1538 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:16.192008 1493 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.192338 1557 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:16.192570 1493 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.193181 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.193532 1231 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.194523 1493 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:16.197474 31881 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:16.198498 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:16.185452 1559 log.cc:927] Time spent T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20250831 07:59:16.181834 940 raft_consensus.cc:695] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:16.185590 1253 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:16.181700 1547 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.206496 1558 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:16.206827 1570 raft_consensus.cc:2802] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:16.207993 1447 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.207463 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.208755 1116 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:16.209618 1505 consensus_queue.cc:260] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.209995 1495 raft_consensus.cc:3053] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:16.210512 1525 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:16.211079 31877 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:16.265911 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.266664 31877 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:16.212316 1548 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.205657 1493 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.213956 1231 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:16.252390 1116 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.227751 1531 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.268985 1521 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.219807 1538 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.185452 1565 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:16.235198 1570 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:16.240841 1569 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.123s
I20250831 07:59:16.333021 31879 raft_consensus.cc:1238] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:16.352202 31880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 190) took 1286 ms. Trace:
I20250831 07:59:16.354831 31880 rpcz_store.cc:276] 0831 07:59:15.065650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:15.065791 (+ 141us) service_pool.cc:225] Handling call
0831 07:59:16.352016 (+1286225us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b5400298bd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:16.352129 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":645437312}
W20250831 07:59:16.355939 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:16.360150 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 85449.6us
I20250831 07:59:16.330798 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.332716 31805 raft_consensus.cc:1238] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:16.252178 1495 raft_consensus.cc:738] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:16.331708 940 consensus_queue.cc:237] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.241972 1447 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.283622 1493 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:16.281853 1523 consensus_queue.cc:237] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.328835 1116 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.350520 1244 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.386910 32471 raft_consensus.cc:2366] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6.
I20250831 07:59:16.339604 1531 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.367614 31882 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:16.368518 31874 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:16.255764 1316 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.369911 1439 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.368543 31775 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:16.372776 1495 consensus_queue.cc:260] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.375877 1396 consensus_queue.cc:1046] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250831 07:59:16.380196 1539 log.cc:927] Time spent T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.523s user 0.005s sys 0.000s
I20250831 07:59:16.388896 32465 raft_consensus.cc:2366] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6.
I20250831 07:59:16.389137 1548 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:16.361418 1570 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:16.378566 31849 connection.cc:232] RPC call timeout handler was delayed by -1.2542087150000043s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1751s for a response
I20250831 07:59:16.393870 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:16.280884 31881 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:16.404471 1569 consensus_queue.cc:1035] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:59:16.406407 1447 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:16.409206 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.352902 1521 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:16.419723 31874 raft_consensus.cc:1215] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[2.5-2.5] Dedup: 2.5->[]
I20250831 07:59:16.420486 31874 raft_consensus.cc:1238] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20250831 07:59:16.413889 31849 connection.cc:232] RPC call timeout handler was delayed by -1.2038087150000365s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1807s for a response
I20250831 07:59:16.416662 31881 raft_consensus.cc:1273] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:59:16.412768 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:16.421397 31849 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.521s (SENT) [suppressed 37 similar messages]
I20250831 07:59:16.432550 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:16.433197 1573 raft_consensus.cc:2802] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:16.433538 1116 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:16.433633 1573 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:16.434175 1573 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:16.435053 1438 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:16.443445 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.444595 31882 raft_consensus.cc:1273] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:16.419636 1495 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:16.453197 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.412420 1439 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.457226 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.457998 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:16.458351 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:16.458928 31728 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:16.461035 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.463469 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.458534 31734 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:16.464947 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.467973 31728 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:16.462081 31877 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:16.470048 32436 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:16.469543 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.472648 31730 raft_consensus.cc:2391] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:16.475737 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.476611 31731 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:16.477175 31877 raft_consensus.cc:1273] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250831 07:59:16.498781 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.472051 1244 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:16.510299 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.412987 1316 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.516597 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.517323 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.515762 31730 raft_consensus.cc:2466] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:16.518033 31877 raft_consensus.cc:1273] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:16.473572 1493 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:16.523686 1316 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:16.518893 1577 consensus_queue.cc:1035] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.016s
I20250831 07:59:16.539966 1538 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:16.527571 1508 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:16.541169 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:16.541942 31875 raft_consensus.cc:2466] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:16.542131 1531 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:16.543964 1566 log.cc:927] Time spent T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.131s user 0.001s sys 0.000s
I20250831 07:59:16.543843 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.544493 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:16.545235 31883 raft_consensus.cc:2391] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:16.545747 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:16.544518 31877 raft_consensus.cc:3053] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:16.548503 31877 raft_consensus.cc:738] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:16.550129 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.486948 1548 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:16.554350 31882 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:16.555224 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:16.557647 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:16.558928 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 153452us
I20250831 07:59:16.561476 31733 raft_consensus.cc:3053] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:16.561839 31733 raft_consensus.cc:738] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:16.523945 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:16.568859 31882 raft_consensus.cc:1273] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:16.562683 31733 consensus_queue.cc:260] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.541019 1385 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.549207 31877 consensus_queue.cc:260] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.574239 32467 raft_consensus.cc:1238] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250831 07:59:16.715150 1495 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:16.649241 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.722071 1495 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.592466 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:16.663969 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 242601us
I20250831 07:59:16.724663 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1676) took 1277 ms. Trace:
W20250831 07:59:16.169667 1469 consensus_queue.cc:1162] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a0d845d01c8c4894895f66e3beae298c" responder_term: 6 status { last_received { term: 3 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 5. Current term is 6. Ops: [5.6-5.6]" } } last_received_current_leader { term: 0 index: 0 } }
W20250831 07:59:16.598309 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.725107 31725 rpcz_store.cc:276] 0831 07:59:15.447263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:15.542299 (+ 95036us) service_pool.cc:225] Handling call
0831 07:59:15.544381 (+ 2082us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:15.577397 (+ 33016us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:16.724461 (+1147064us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b540038b8d0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:16.724580 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3989248}
W20250831 07:59:16.727111 31849 leader_election.cc:336] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.615s (SENT)
W20250831 07:59:16.728802 31849 leader_election.cc:336] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.697s (SENT)
W20250831 07:59:16.725240 32464 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47886 (request call id 393) took 2064 ms (client timeout 2666 ms). Trace:
W20250831 07:59:16.730487 31849 connection.cc:232] RPC call timeout handler was delayed by -1.1292265960000805s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16110000000000002s for a response
W20250831 07:59:16.730391 32464 rpcz_store.cc:269] 0831 07:59:14.660696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:14.666509 (+ 5813us) service_pool.cc:225] Handling call
0831 07:59:16.725033 (+2058524us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b54005bf754. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:16.725183 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55881984}
I20250831 07:59:16.642984 32462 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38366 (request call id 429) took 1581 ms. Trace:
W20250831 07:59:16.630518 1576 log.cc:927] Time spent T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.198s user 0.001s sys 0.000s
I20250831 07:59:16.540004 1439 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:16.732872 1385 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:16.541054 1508 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:16.726970 1487 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:16.643512 32463 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38366 (request call id 433) took 1043 ms. Trace:
I20250831 07:59:16.733531 31877 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:16.733016 1269 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:16.591967 1579 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.573112 1548 raft_consensus.cc:695] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:16.734325 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.600554 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:16.541285 1547 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:16.620002 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:16.739090 1378 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:16.736374 31849 connection.cc:232] RPC call timeout handler was delayed by -1.1219087150000178s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1898s for a response
I20250831 07:59:16.601722 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:16.738240 31733 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:16.760219 31849 leader_election.cc:336] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.893s (SENT)
I20250831 07:59:16.589213 31734 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:16.677008 1466 raft_consensus.cc:3053] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 3 LEADER]: Stepping down as leader of term 3
W20250831 07:59:16.605590 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.762634 31880 raft_consensus.cc:1238] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:16.594722 1570 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:16.757154 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.757999 32471 raft_consensus.cc:2408] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20250831 07:59:16.632953 1533 log.cc:927] Time spent T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.962s user 0.005s sys 0.003s
I20250831 07:59:16.627480 1477 consensus_queue.cc:1035] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.125s
I20250831 07:59:16.725204 1495 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:16.635998 1575 consensus_queue.cc:1035] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.118s
W20250831 07:59:16.763541 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.754338 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:16.765095 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.498112 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.582216 1573 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.782938 1573 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:16.784256 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 223565us
I20250831 07:59:16.793390 31877 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:16.736640 32462 rpcz_store.cc:276] 0831 07:59:15.062101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:15.065852 (+ 3751us) service_pool.cc:225] Handling call
0831 07:59:16.642750 (+1576898us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b540066bf54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:16.642899 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":443831680}
I20250831 07:59:16.762218 1466 raft_consensus.cc:738] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:16.757463 1579 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.751775 32463 rpcz_store.cc:276] 0831 07:59:15.599651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:15.599826 (+ 175us) service_pool.cc:225] Handling call
0831 07:59:16.643325 (+1043499us) spinlock_profiling.cc:243] Waited 48.1 ms on lock 0x7b54004c13d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:16.643452 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110563584}
I20250831 07:59:16.804733 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.805143 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:16.805812 32471 raft_consensus.cc:2391] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:16.805874 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.806149 1579 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:16.765559 1378 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:16.806859 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.806916 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.808284 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.808367 31698 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:16.808849 1378 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:16.809425 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.810274 1523 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:16.811048 1523 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:16.811389 31731 raft_consensus.cc:1238] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:16.811547 1523 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:16.813004 31731 raft_consensus.cc:1273] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:16.814390 31731 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250831 07:59:16.815874 31733 raft_consensus.cc:1273] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:16.816135 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.817417 31732 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250831 07:59:16.818027 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:16.819336 1570 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:16.821408 32462 raft_consensus.cc:1238] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:16.823194 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:16.824038 31806 raft_consensus.cc:2366] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5.
W20250831 07:59:16.824867 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 234059us
I20250831 07:59:16.828894 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:16.829936 31808 raft_consensus.cc:2391] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:16.831090 31734 raft_consensus.cc:1273] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:16.832398 31727 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:16.832554 32471 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250831 07:59:16.834486 31728 raft_consensus.cc:1238] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:16.810457 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:16.836012 31730 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:16.839053 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:16.841982 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:16.842900 31801 raft_consensus.cc:2391] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:16.815630 31731 raft_consensus.cc:1273] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250831 07:59:16.843225 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 47685.2us
I20250831 07:59:16.844805 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:16.844553 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:16.848429 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:16.849114 31773 leader_election.cc:400] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6.
I20250831 07:59:16.848978 31725 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:16.849862 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:16.850373 32436 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:16.851473 1306 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:16.852365 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:16.854981 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.831640 1570 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.856114 1570 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.858055 1570 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:16.854594 1378 consensus_queue.cc:1046] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20250831 07:59:16.845052 1439 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.861099 1044 raft_consensus.cc:3053] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Stepping down as leader of term 3
W20250831 07:59:16.861912 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.863610 31809 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250831 07:59:16.864643 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:16.865371 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.865705 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:16.865898 31876 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 6
I20250831 07:59:16.867296 1407 consensus_queue.cc:1046] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:59:16.868513 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:16.876318 31730 raft_consensus.cc:2466] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
W20250831 07:59:16.912266 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 45348us
I20250831 07:59:16.846547 1495 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:16.844516 1583 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.908082 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:16.922242 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:16.912309 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 50283.2us
I20250831 07:59:16.914264 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:16.853951 1306 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:16.923709 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:16.917871 31806 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:16.861466 1044 raft_consensus.cc:738] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:16.928954 31773 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:16.919283 31876 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:16.846431 1548 consensus_queue.cc:237] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.925026 31801 raft_consensus.cc:1238] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:16.917407 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:16.925418 31809 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:16.936578 31806 raft_consensus.cc:1238] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:16.937750 31809 raft_consensus.cc:1273] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:16.937853 31800 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:16.939579 31800 raft_consensus.cc:1238] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 07:59:16.861467 1439 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:16.945323 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:16.943984 31801 raft_consensus.cc:1238] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 07:59:16.947213 31773 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:16.950181 1385 consensus_queue.cc:1035] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250831 07:59:16.951195 1582 log.cc:927] Time spent T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20250831 07:59:16.952349 31801 raft_consensus.cc:1273] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:16.953065 31849 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:16.955142 31806 raft_consensus.cc:1238] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:16.956105 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:16.969115 31806 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:16.970726 31773 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:16.975399 31807 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:16.956300 1547 consensus_queue.cc:1035] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20250831 07:59:16.977958 31807 raft_consensus.cc:1238] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:16.933974 1570 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:16.927265 1306 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:16.919451 1477 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20250831 07:59:16.980590 31848 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:16.942559 940 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:16.984406 1385 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:16.991060 31773 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:16.922622 1495 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.913440 1466 consensus_queue.cc:260] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:16.993875 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1680) took 3033 ms. Trace:
I20250831 07:59:16.930171 1474 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20250831 07:59:16.952541 1044 consensus_queue.cc:260] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:16.996052 31773 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.017437 31773 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:17.017647 31877 raft_consensus.cc:1273] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:16.925819 1583 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:17.005587 31806 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:17.021377 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.021421 31881 raft_consensus.cc:1273] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 07:59:17.022451 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:16.997083 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:16.980448 1589 raft_consensus.cc:2747] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:17.014281 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:17.025024 31876 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:17.025856 31773 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.026479 31806 raft_consensus.cc:1238] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:17.002327 31849 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:17.027012 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 156018us
I20250831 07:59:17.027323 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:17.005944 31803 rpcz_store.cc:276] 0831 07:59:13.960645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:15.055979 (+1095334us) service_pool.cc:225] Handling call
0831 07:59:15.160725 (+104746us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:15.394109 (+233384us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:15.394137 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:15.394192 (+ 55us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:15.394237 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:16.993655 (+1599418us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b540003a7d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:16.993704 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:16.993824 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":536332544}
I20250831 07:59:17.029069 32434 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:16.956805 1378 consensus_queue.cc:1035] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.058s
I20250831 07:59:17.031620 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:17.032224 32467 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:17.004900 1407 raft_consensus.cc:2747] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:16.996052 1570 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:17.040875 1495 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:17.041661 1044 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:17.042889 31803 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620"
dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:39082
W20250831 07:59:17.053225 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:17.053620 31803 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 07:59:17.054024 31803 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:17.054056 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1625) took 1376 ms. Trace:
I20250831 07:59:17.054515 31878 rpcz_store.cc:276] 0831 07:59:15.678059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:15.678193 (+ 134us) service_pool.cc:225] Handling call
0831 07:59:15.678481 (+ 288us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:15.987933 (+309452us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:15.987981 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:15.988043 (+ 62us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:15.988094 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:17.053851 (+1065757us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b54001ed7d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:17.053896 (+ 45us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:17.054004 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":711070976}
I20250831 07:59:17.036073 1472 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:17.056363 1334 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:17.056883 1334 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:17.057962 1455 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20250831 07:59:17.058846 1334 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:17.049700 31773 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:16.997133 1585 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:17.031569 1586 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:17.036734 940 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:17.050537 1477 raft_consensus.cc:2747] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:17.050894 31805 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:16.997014 1487 raft_consensus.cc:2802] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:16.992976 1547 consensus_queue.cc:1035] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20250831 07:59:17.063535 32467 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:17.065804 31773 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:17.069535 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 67263.1us
I20250831 07:59:17.061254 1573 raft_consensus.cc:2747] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:17.084687 31803 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:17.090427 31849 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.089939 1458 raft_consensus.cc:2747] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:17.093482 1334 consensus_queue.cc:1035] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:59:17.093662 31773 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:17.094161 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 237) took 1018 ms. Trace:
I20250831 07:59:17.094903 31773 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:17.094935 1334 raft_consensus.cc:2747] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:17.096051 1334 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:17.097114 940 consensus_queue.cc:1035] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:59:17.097445 32467 raft_consensus.cc:1238] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:17.095209 1589 consensus_queue.cc:1035] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:17.098124 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.101900 31800 raft_consensus.cc:2391] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:17.066406 1487 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:17.094548 31804 rpcz_store.cc:276] 0831 07:59:16.075562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:16.093735 (+ 18173us) service_pool.cc:225] Handling call
0831 07:59:16.139514 (+ 45779us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:16.186331 (+ 46817us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:17.093934 (+907603us) spinlock_profiling.cc:243] Waited 43.6 ms on lock 0x7b540015a754. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:17.094107 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":100343168}
I20250831 07:59:17.098973 31773 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.105316 31804 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250831 07:59:17.105523 31773 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:17.067318 1586 raft_consensus.cc:695] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:17.103475 31848 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:17.103855 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.113283 31773 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:17.113595 31800 raft_consensus.cc:1238] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 07:59:17.115123 31800 raft_consensus.cc:1273] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:59:17.073935 1438 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:17.066104 1466 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:17.118495 31773 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:17.119180 31803 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:17.121202 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.106622 1407 raft_consensus.cc:2747] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:17.130143 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 99091.1us
I20250831 07:59:17.103698 1570 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:17.112118 1487 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:17.120350 1458 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:17.134172 32434 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:17.102025 1523 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:17.122367 1575 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:17.124931 1385 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:17.144879 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:17.145030 1385 raft_consensus.cc:695] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:17.145884 1385 consensus_queue.cc:237] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:17.127496 1442 raft_consensus.cc:2802] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:17.103446 1573 consensus_queue.cc:1035] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20250831 07:59:17.148662 1442 raft_consensus.cc:695] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:17.149777 1442 consensus_queue.cc:237] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:17.138842 1508 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:17.152099 32465 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:17.156231 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.157667 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.158841 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:17.166463 1572 log.cc:927] Time spent T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.647s user 0.001s sys 0.003s
I20250831 07:59:17.170503 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39068 (request call id 597) took 1017 ms. Trace:
I20250831 07:59:17.138703 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.129961 1438 raft_consensus.cc:2747] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:17.102972 1454 consensus_queue.cc:1046] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20250831 07:59:17.137869 940 consensus_queue.cc:1035] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.004s
I20250831 07:59:17.143008 1523 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:17.116988 1378 consensus_queue.cc:1035] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20250831 07:59:17.138731 1458 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:17.120234 1477 consensus_queue.cc:1035] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.020s
I20250831 07:59:17.159695 31800 raft_consensus.cc:2391] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:17.116719 1472 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20250831 07:59:17.171487 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 183) took 1005 ms. Trace:
I20250831 07:59:17.184700 31729 rpcz_store.cc:276] 0831 07:59:16.165749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:16.167016 (+ 1267us) service_pool.cc:225] Handling call
0831 07:59:16.167333 (+ 317us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:16.178565 (+ 11232us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:16.178593 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:16.179090 (+ 497us) log.cc:844] Serialized 26 byte log entry
0831 07:59:16.231479 (+ 52389us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:16.231521 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:16.231592 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:17.165714 (+934122us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:17.171112 (+ 5398us) raft_consensus.cc:1683] finished
0831 07:59:17.171137 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:17.171416 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15150,"raft.queue_time_us":13386,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":5711,"spinlock_wait_cycles":29818112,"thread_start_us":52053,"threads_started":1,"wal-append.queue_time_us":275267}
I20250831 07:59:17.174475 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:17.187250 31733 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:17.171052 31802 rpcz_store.cc:276] 0831 07:59:16.153147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:16.153736 (+ 589us) service_pool.cc:225] Handling call
0831 07:59:17.169680 (+1015944us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b54002f0654. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f81cd0 00007fdce8005eea 00007fdcecf5e60e 00007fdce65fbcc7 00007fdce65fbc52 00007fdce65fbb81 00007fdce65fbafc 00007fdce65fadb2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:17.170420 (+ 740us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1639366400}
I20250831 07:59:17.179484 1454 raft_consensus.cc:3053] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:17.190002 1454 raft_consensus.cc:738] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:17.180703 1458 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:17.196317 31699 leader_election.cc:336] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 2.113s (SENT)
I20250831 07:59:17.196897 31699 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:17.200040 1303 raft_consensus.cc:2747] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:17.181823 1585 raft_consensus.cc:2747] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:17.177906 31808 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:17.205456 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.207069 31801 raft_consensus.cc:1273] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:17.208105 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.208297 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:17.209396 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:17.209650 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.181013 31803 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:17.215170 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.215504 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.216315 31802 raft_consensus.cc:2374] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4. Re-sending same reply.
I20250831 07:59:17.216567 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.211942 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.219281 31802 raft_consensus.cc:1273] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 07:59:17.220001 31808 raft_consensus.cc:1811] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Handling vote request from an unknown peer a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.223978 31808 raft_consensus.cc:2408] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20250831 07:59:17.226235 31699 leader_election.cc:336] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 2.141s (SENT)
I20250831 07:59:17.226871 31699 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:17.191020 1454 consensus_queue.cc:260] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:17.172435 1508 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:17.170871 1306 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:17.282704 1492 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.016s sys 0.017s
I20250831 07:59:17.283282 32470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47886 (request call id 398) took 1315 ms. Trace:
W20250831 07:59:17.283576 1492 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.775s user 0.016s sys 0.017s
I20250831 07:59:17.283914 32470 rpcz_store.cc:276] 0831 07:59:15.967762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:15.967956 (+ 194us) service_pool.cc:225] Handling call
0831 07:59:15.984858 (+ 16902us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:16.020941 (+ 36083us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:17.282957 (+1262016us) spinlock_profiling.cc:243] Waited 35.2 ms on lock 0x7b54003ba954. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:17.283203 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":80866560}
I20250831 07:59:17.285753 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1206) took 1182 ms. Trace:
I20250831 07:59:17.286268 31879 rpcz_store.cc:276] 0831 07:59:16.103101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:16.104054 (+ 953us) service_pool.cc:225] Handling call
0831 07:59:16.332855 (+228801us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:16.354370 (+ 21515us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:17.285508 (+931138us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b5400058550. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:17.285676 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":525388544}
I20250831 07:59:17.290144 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38366 (request call id 431) took 1691 ms. Trace:
W20250831 07:59:17.298120 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 174109us
W20250831 07:59:17.300225 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 166121us
I20250831 07:59:17.246198 1454 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:17.308753 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:17.306135 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:17.305439 31773 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:17.319005 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.172571 1586 consensus_queue.cc:237] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:17.322551 32434 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:17.252833 32462 raft_consensus.cc:1238] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
W20250831 07:59:17.324602 31699 leader_election.cc:336] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 2.240s (SENT)
I20250831 07:59:17.331528 31699 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:17.335362 31733 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:17.337160 31803 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8.
I20250831 07:59:17.338318 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:17.341349 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 270899us
I20250831 07:59:17.341838 31773 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:17.320066 1508 raft_consensus.cc:2747] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:17.330601 1477 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:17.353401 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:17.359125 32467 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:17.342113 31849 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:17.360560 31773 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:17.326826 1472 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:17.362828 31849 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.347201 1303 raft_consensus.cc:2747] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:17.348426 1454 raft_consensus.cc:2747] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:17.372038 32470 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20250831 07:59:17.377300 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 52734.6us
I20250831 07:59:17.379413 31805 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:17.350229 1018 raft_consensus.cc:2802] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:17.325743 1558 consensus_queue.cc:1035] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20250831 07:59:17.330935 1589 consensus_queue.cc:1046] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20250831 07:59:17.363883 1477 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:17.384819 31800 raft_consensus.cc:1238] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:17.373929 1044 raft_consensus.cc:2747] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:17.372179 1385 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:17.393345 31849 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.382932 1334 consensus_queue.cc:1046] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
W20250831 07:59:17.400966 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:17.290695 32469 rpcz_store.cc:276] 0831 07:59:15.598975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:15.599132 (+ 157us) service_pool.cc:225] Handling call
0831 07:59:15.599556 (+ 424us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:15.687056 (+ 87500us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:15.690743 (+ 3687us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:15.887558 (+196815us) log.cc:844] Serialized 26 byte log entry
0831 07:59:16.093086 (+205528us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:16.102125 (+ 9039us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:16.733912 (+631787us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b1800456be0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce5069786 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89
0831 07:59:17.289715 (+555803us) spinlock_profiling.cc:243] Waited 75.3 ms on lock 0x7b5400144e54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:17.289794 (+ 79us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:17.289865 (+ 71us) raft_consensus.cc:1683] finished
0831 07:59:17.289886 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:17.290071 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2332,"raft.queue_time_us":6878,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":13345,"spinlock_wait_cycles":198739200,"thread_start_us":126457,"threads_started":1,"wal-append.queue_time_us":195421,"wal-append.run_cpu_time_us":1711,"wal-append.run_wall_time_us":1089493}
W20250831 07:59:17.405966 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 69146.4us
I20250831 07:59:17.385612 1508 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:17.407474 32436 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:17.407996 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:17.409381 31729 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:17.409799 31849 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.387442 1477 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:17.410960 1595 log.cc:927] Time spent T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20250831 07:59:17.403525 1240 consensus_queue.cc:1035] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.013s
I20250831 07:59:17.372541 1472 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:17.417260 1306 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:17.417399 1044 raft_consensus.cc:2747] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:17.418720 1472 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:17.430126 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.431771 31849 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.431969 31805 raft_consensus.cc:1273] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:17.409183 1261 raft_consensus.cc:3053] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:17.395093 1018 raft_consensus.cc:695] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:17.436219 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.438835 31729 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:17.438249 31849 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.392073 1385 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:17.438184 1378 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:17.433030 1240 raft_consensus.cc:2747] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:17.408696 1600 consensus_queue.cc:1035] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20250831 07:59:17.435523 1261 raft_consensus.cc:738] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:17.447957 1438 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:17.448108 1261 consensus_queue.cc:260] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:17.449532 1261 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:17.446538 1385 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:17.453073 1508 raft_consensus.cc:695] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:17.445168 31849 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.429687 1458 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:17.472414 1458 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:17.472684 1367 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:17.473277 31802 raft_consensus.cc:1238] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:17.473312 1367 raft_consensus.cc:2747] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:17.475032 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.476936 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.477538 31809 raft_consensus.cc:2374] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3. Re-sending same reply.
I20250831 07:59:17.478475 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.479228 1487 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:17.482261 31804 raft_consensus.cc:1273] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 07:59:17.479674 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.460608 1492 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:17.475487 32470 raft_consensus.cc:1238] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:17.484899 31849 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.463672 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.485033 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:17.481184 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 73813.2us
I20250831 07:59:17.483629 31809 raft_consensus.cc:2374] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3. Re-sending same reply.
W20250831 07:59:17.490542 1609 log.cc:927] Time spent T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20250831 07:59:17.483047 1606 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:17.491202 1606 raft_consensus.cc:695] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:17.491775 1018 consensus_queue.cc:237] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:17.492071 1606 consensus_queue.cc:237] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:17.492691 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.495297 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:17.496363 1600 consensus_queue.cc:1035] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:17.496191 1492 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:17.497954 1601 consensus_queue.cc:1046] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:59:17.499554 1492 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:17.501608 1439 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:17.454038 1508 consensus_queue.cc:237] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:17.502094 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:17.502210 31809 raft_consensus.cc:3053] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:17.502655 31809 raft_consensus.cc:738] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:17.503532 31809 consensus_queue.cc:260] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:17.506021 31809 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:17.512483 32436 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:17.515566 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:17.515949 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:17.516335 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:17.516731 31729 raft_consensus.cc:2391] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:17.517082 31730 raft_consensus.cc:2391] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 6: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:17.517525 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:17.517686 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:17.518337 31725 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:17.517095 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:17.519163 31734 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:17.519438 31882 raft_consensus.cc:2391] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:17.522734 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:17.523655 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:17.525283 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:17.526084 31730 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:17.549880 31809 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20250831 07:59:17.550310 31809 pending_rounds.cc:85] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 6
I20250831 07:59:17.550515 31809 pending_rounds.cc:107] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250831 07:59:17.550899 31809 raft_consensus.cc:2887] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:17.534832 31729 pending_rounds.cc:85] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 1
I20250831 07:59:17.560958 31729 pending_rounds.cc:107] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250831 07:59:17.561280 31729 raft_consensus.cc:2887] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:17.530469 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:17.479794 1385 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:17.562973 1032 raft_consensus.cc:3053] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:17.563411 1032 raft_consensus.cc:738] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:17.564347 31732 raft_consensus.cc:1238] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:17.564246 1032 consensus_queue.cc:260] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:17.565340 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:17.567528 1275 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:17.568625 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.569748 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.570461 31800 raft_consensus.cc:2391] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:17.571167 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.530573 1528 consensus_queue.cc:1035] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20250831 07:59:17.572481 31809 raft_consensus.cc:1238] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:17.526590 1306 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:17.573660 1306 raft_consensus.cc:695] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:17.562052 31731 raft_consensus.cc:2391] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:17.574538 1306 consensus_queue.cc:237] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:17.581604 1438 raft_consensus.cc:2747] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:17.567600 1487 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:17.567667 31725 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:17.586663 31803 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
W20250831 07:59:17.587039 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:17.589766 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:17.590621 31847 leader_election.cc:400] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6.
I20250831 07:59:17.565407 1032 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:17.518563 1600 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:17.605650 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:17.605787 31876 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 07:59:17.594250 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:17.594494 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:17.595244 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:17.608460 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.609222 31803 raft_consensus.cc:2374] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6. Re-sending same reply.
I20250831 07:59:17.597558 31849 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.591892 31847 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:17.619009 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.619321 32467 raft_consensus.cc:1238] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250831 07:59:17.619635 31809 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:17.622496 31878 raft_consensus.cc:1238] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20250831 07:59:17.623533 31849 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:17.552366 1439 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:17.624727 31849 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:17.625530 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:17.626240 31880 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:17.627429 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:17.628159 31883 raft_consensus.cc:3053] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:17.628517 31883 raft_consensus.cc:738] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:17.640287 31803 raft_consensus.cc:1238] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:17.641430 31734 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:17.649662 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:17.650516 32471 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:17.650916 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:17.654284 31734 raft_consensus.cc:2391] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:17.629379 31883 consensus_queue.cc:260] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:17.657449 31883 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:17.658705 31734 pending_rounds.cc:85] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 5
I20250831 07:59:17.656785 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.670238 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.670962 31807 raft_consensus.cc:2374] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4. Re-sending same reply.
I20250831 07:59:17.671087 31809 raft_consensus.cc:2374] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8. Re-sending same reply.
I20250831 07:59:17.681279 31734 pending_rounds.cc:107] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250831 07:59:17.681514 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:17.672878 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:17.684753 31880 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 5
I20250831 07:59:17.673883 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:17.685091 31809 raft_consensus.cc:2391] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:17.685346 32471 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:17.686528 31734 raft_consensus.cc:2887] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:17.693207 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:17.697957 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:17.605003 1612 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20250831 07:59:17.661386 1253 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:16.779989 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:59:17.661116 1487 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:17.707777 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.710311 1477 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:17.717209 32468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38366 (request call id 430) took 2121 ms. Trace:
I20250831 07:59:17.724172 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:17.691653 1264 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:17.725456 1585 raft_consensus.cc:3053] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:17.656972 1439 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:17.672736 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:17.739391 31802 raft_consensus.cc:2374] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8. Re-sending same reply.
I20250831 07:59:17.667330 1520 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:17.751385 31883 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:18.005121 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.005972 31883 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:17.755791 31804 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:17.772204 32471 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
W20250831 07:59:17.867383 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:17.874207 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:17.927764 31880 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:17.727921 1466 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:17.708652 1447 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:17.800902 1603 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:17.680012 1367 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:17.936023 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:17.936780 32468 rpcz_store.cc:276] 0831 07:59:15.595412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:15.595533 (+ 121us) service_pool.cc:225] Handling call
0831 07:59:16.218259 (+622726us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:16.229400 (+ 11141us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:16.229445 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:16.355491 (+126046us) log.cc:844] Serialized 26 byte log entry
0831 07:59:16.558518 (+203027us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:16.558586 (+ 68us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:16.558700 (+ 114us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:16.952020 (+393320us) raft_consensus.cc:1683] finished
0831 07:59:16.952041 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:17.716870 (+764829us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b540066b2d0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:17.717060 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12751,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":4108,"spinlock_wait_cycles":1427246976,"thread_start_us":169526,"threads_started":1,"wal-append.queue_time_us":377229}
W20250831 07:59:17.953713 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 341440us
W20250831 07:59:17.956269 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 364484us
I20250831 07:59:17.969444 31775 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:18.053458 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 84112.3us
W20250831 07:59:17.985003 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 254388us
I20250831 07:59:17.986827 31800 raft_consensus.cc:2366] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6.
I20250831 07:59:18.056787 31699 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:17.989030 1477 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:17.992705 32464 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:17.990088 1585 raft_consensus.cc:738] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:18.033891 31847 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:18.071076 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:18.004241 1520 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:18.077731 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 43936.9us
I20250831 07:59:17.943332 1253 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.080595 31847 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:18.084677 32468 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:18.067337 1458 raft_consensus.cc:2747] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:17.992859 1264 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:18.091104 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:18.094100 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.085455 1520 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:18.101557 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:18.095155 1616 log.cc:927] Time spent T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.243s user 0.001s sys 0.000s
I20250831 07:59:18.059130 1466 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.067028 1400 consensus_queue.cc:1046] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.027s
W20250831 07:59:18.101693 31731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.000s sys 0.017s
I20250831 07:59:18.061031 1447 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.069458 1603 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.111747 1439 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:18.113085 1253 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:18.109349 1585 consensus_queue.cc:260] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.099444 1385 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:18.116928 31731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s user 0.000s sys 0.017s
W20250831 07:59:18.117282 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:18.118451 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 61788.5us
W20250831 07:59:18.135161 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:18.094280 1367 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 07:59:18.114691 1275 consensus_meta.cc:287] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.546s user 0.007s sys 0.002s
I20250831 07:59:18.088675 1018 consensus_queue.cc:1046] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
W20250831 07:59:18.131649 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 75909.4us
I20250831 07:59:18.135916 31699 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:18.130204 1044 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:18.130029 1625 raft_consensus.cc:2747] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:18.146271 31726 raft_consensus.cc:1273] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:18.135882 1585 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:18.136358 1624 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:18.137102 1367 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:18.170933 31881 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.011s sys 0.010s
I20250831 07:59:18.174136 1508 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:18.174721 1439 consensus_queue.cc:1035] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:59:18.174841 1458 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250831 07:59:18.175964 31775 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:18.179177 31881 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s user 0.012s sys 0.011s
I20250831 07:59:18.144320 1400 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:18.145623 1385 raft_consensus.cc:3053] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:18.147877 1495 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:18.202118 1600 consensus_meta.cc:287] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.580s user 0.009s sys 0.009s
W20250831 07:59:18.187920 1621 log.cc:927] Time spent T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
W20250831 07:59:18.192318 1622 log.cc:927] Time spent T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20250831 07:59:18.183485 1625 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:18.192973 1264 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:18.170372 1018 consensus_queue.cc:1230] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer 7ecb257174554a26b2c35ad04b0eabae log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250831 07:59:18.168049 1303 consensus_queue.cc:1035] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20250831 07:59:18.169901 1044 raft_consensus.cc:695] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:18.206568 1466 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:18.178158 1367 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:18.202749 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:18.205485 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:18.177914 1624 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.178840 1523 consensus_queue.cc:1046] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
W20250831 07:59:18.207720 31849 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.006s sys 0.015s
W20250831 07:59:18.298540 31849 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.673s user 0.007s sys 0.015s
W20250831 07:59:18.299132 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 674437us
I20250831 07:59:18.299837 31849 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:18.301559 31849 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:18.207878 31848 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.007s sys 0.008s
W20250831 07:59:18.304006 31848 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.644s user 0.007s sys 0.008s
I20250831 07:59:18.265820 1600 raft_consensus.cc:2747] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 07:59:18.206331 1623 log.cc:927] Time spent T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20250831 07:59:18.291033 31807 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:18.292044 31727 raft_consensus.cc:2408] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250831 07:59:18.320009 32435 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:18.211530 31775 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:18.185616 1508 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:18.304364 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 654567us
W20250831 07:59:18.231038 31733 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.006s sys 0.023s
W20250831 07:59:18.322510 31733 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.664s user 0.006s sys 0.023s
I20250831 07:59:18.323449 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:18.324744 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:18.325353 31733 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:18.323469 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:18.270115 1447 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:18.265496 1495 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:18.332422 1640 log.cc:927] Time spent T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20250831 07:59:18.334268 1508 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:18.340411 1603 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:18.340632 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:18.341228 31731 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:18.343107 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:18.344467 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.345537 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.346541 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:18.347440 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.228853 1400 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.350230 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.350808 31877 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:18.351204 1400 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:18.214149 1385 raft_consensus.cc:738] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:18.351675 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:18.352198 31877 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:18.352496 1385 consensus_queue.cc:260] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.353665 1385 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:18.344125 1044 consensus_queue.cc:237] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.274292 1625 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.359002 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:18.359407 1624 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:18.213871 1261 raft_consensus.cc:2747] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:18.360484 31731 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:18.229411 1477 consensus_queue.cc:1046] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.132s
I20250831 07:59:18.362213 1477 raft_consensus.cc:3053] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:18.362540 1477 raft_consensus.cc:738] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:18.364859 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 153450us
I20250831 07:59:18.364733 1520 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.365816 31775 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:18.366930 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:18.367964 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 48039.6us
I20250831 07:59:18.369087 31847 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:18.370683 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:18.371241 31807 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:18.378311 31847 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:18.390048 31733 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:18.391247 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.392313 32435 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:18.403623 32434 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:18.399765 31773 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:18.405050 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.405548 31773 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:18.387800 31877 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:18.406509 31773 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:18.406957 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.394552 31728 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:18.380992 1624 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:18.316507 1547 raft_consensus.cc:2802] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:18.410876 31807 raft_consensus.cc:3053] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Stepping down as leader of term 5
W20250831 07:59:18.411007 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:18.404307 1625 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:18.427304 1487 consensus_meta.cc:287] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.580s user 0.011s sys 0.000s
I20250831 07:59:18.385164 1458 consensus_queue.cc:1035] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20250831 07:59:18.426285 1508 raft_consensus.cc:2802] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:18.380353 1628 consensus_queue.cc:1046] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20250831 07:59:18.363297 1477 consensus_queue.cc:260] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.366684 1520 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:18.372036 1498 raft_consensus.cc:2747] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:18.423794 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:18.381366 1439 raft_consensus.cc:2747] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 07:59:18.404372 1613 log.cc:927] Time spent T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20250831 07:59:18.381443 1261 raft_consensus.cc:2802] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:18.385066 1264 raft_consensus.cc:2802] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:18.379945 1634 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:18.396025 1626 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:18.427024 1400 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:18.405624 31881 raft_consensus.cc:2374] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6. Re-sending same reply.
I20250831 07:59:18.431741 31848 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:18.483373 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 51786.8us
W20250831 07:59:18.437996 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 294786us
I20250831 07:59:18.430097 31877 raft_consensus.cc:2408] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 07:59:18.454586 31807 raft_consensus.cc:738] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:18.476925 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:18.456915 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 90324.8us
I20250831 07:59:18.494969 31775 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:18.472681 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.496528 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:18.498277 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 91836.1us
I20250831 07:59:18.499054 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.500319 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.501330 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.502291 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:18.502789 31881 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:18.490435 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 113309us
I20250831 07:59:18.509907 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.510262 31699 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:18.460227 1477 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:18.479852 1244 raft_consensus.cc:2802] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:18.457932 1264 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:18.461723 1439 raft_consensus.cc:2802] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:18.465063 1520 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:18.445212 1487 raft_consensus.cc:2747] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:18.526443 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:18.445133 1508 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:18.504063 1458 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:18.430763 1547 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:18.529834 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.461928 1637 raft_consensus.cc:2747] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:18.432996 1624 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.466346 1400 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.523955 1044 raft_consensus.cc:3053] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:18.504184 1303 consensus_queue.cc:1046] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20250831 07:59:18.492336 31807 consensus_queue.cc:260] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.540460 31802 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:18.551864 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:18.556731 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:18.465379 1634 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.528646 1628 raft_consensus.cc:2802] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:18.531147 1264 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:18.545898 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 334281us
I20250831 07:59:18.522969 1627 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
W20250831 07:59:18.560933 1646 log.cc:927] Time spent T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
I20250831 07:59:18.562052 1400 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:18.557839 1044 raft_consensus.cc:738] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:18.562538 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:18.562891 1624 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:18.531796 1654 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:18.563694 1654 raft_consensus.cc:695] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:18.563243 1044 consensus_queue.cc:260] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.564677 1044 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:18.564607 1654 consensus_queue.cc:237] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.566185 1585 consensus_queue.cc:1035] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:59:18.466475 1626 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.568599 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47886 (request call id 400) took 2585 ms. Trace:
I20250831 07:59:18.522984 1649 consensus_queue.cc:1046] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20250831 07:59:18.570034 1649 raft_consensus.cc:3053] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:18.570174 1626 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:18.570623 1649 raft_consensus.cc:738] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:18.567552 1585 raft_consensus.cc:2747] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:18.568684 1648 consensus_queue.cc:1046] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.006s
I20250831 07:59:18.557082 31877 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 6: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:18.576330 1523 raft_consensus.cc:3053] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:18.576725 1523 raft_consensus.cc:738] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:18.577458 1523 consensus_queue.cc:260] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.522158 1447 consensus_queue.cc:1035] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20250831 07:59:18.568867 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:18.551388 1547 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:18.549474 1508 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:18.534062 1520 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:18.549243 1458 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:18.582464 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:18.584408 31881 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:18.556720 31807 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:18.599740 31881 raft_consensus.cc:1238] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20250831 07:59:18.599891 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 104963us
I20250831 07:59:18.608199 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:18.609019 31802 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:18.610726 31802 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20250831 07:59:18.611164 31802 pending_rounds.cc:85] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 5
I20250831 07:59:18.612193 31802 pending_rounds.cc:107] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20250831 07:59:18.612700 31802 raft_consensus.cc:2887] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:18.616734 31807 raft_consensus.cc:1273] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20250831 07:59:18.616890 31775 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:18.617260 31807 pending_rounds.cc:85] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 4
I20250831 07:59:18.615224 1261 raft_consensus.cc:695] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:18.618803 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:18.559720 1628 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:18.627943 31725 raft_consensus.cc:1273] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
W20250831 07:59:18.626487 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 106278us
I20250831 07:59:18.578706 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:18.559723 1625 consensus_queue.cc:1035] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20250831 07:59:18.620505 31807 pending_rounds.cc:107] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20250831 07:59:18.575582 1367 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.628319 31725 pending_rounds.cc:85] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 5
I20250831 07:59:18.637303 31725 pending_rounds.cc:107] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250831 07:59:18.637615 31725 raft_consensus.cc:2887] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:18.624809 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:18.639762 31807 raft_consensus.cc:2887] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:18.641364 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:18.629738 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 130812us
I20250831 07:59:18.606935 1458 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:18.646163 31725 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250831 07:59:18.598697 1520 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:18.626156 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.569461 32466 rpcz_store.cc:276] 0831 07:59:15.983364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:16.133558 (+150194us) service_pool.cc:225] Handling call
0831 07:59:16.661498 (+527940us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:16.661790 (+ 292us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:16.661831 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:16.661896 (+ 65us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:16.661960 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:16.662048 (+ 88us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:18.557961 (+1895913us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b54004c13d0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:18.568535 (+ 10574us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1183031552}
I20250831 07:59:18.606835 1645 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:18.648963 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.648815 1645 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.650249 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:18.650872 31881 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:18.650985 1645 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:18.635242 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:18.632258 1628 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:18.652505 1645 consensus_queue.cc:1035] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:18.626873 1585 raft_consensus.cc:2747] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:18.660266 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:18.618074 1439 raft_consensus.cc:695] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:18.665218 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:18.571575 1649 consensus_queue.cc:260] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.681452 31880 raft_consensus.cc:2408] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250831 07:59:18.656916 1520 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.638178 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:18.625684 1466 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
W20250831 07:59:18.696578 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:18.680671 1469 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:18.705284 1469 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.680171 1668 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:18.702507 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:18.703322 32466 raft_consensus.cc:1273] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20250831 07:59:18.704133 1657 log.cc:927] Time spent T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20250831 07:59:18.687922 1385 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:18.695966 1447 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:18.687556 1303 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:18.723932 1659 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:18.724094 1303 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.724545 1659 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.724902 1634 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:18.723680 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1246) took 1122 ms. Trace:
I20250831 07:59:18.660291 1585 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
W20250831 07:59:18.691334 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 49298.2us
I20250831 07:59:18.734140 31877 raft_consensus.cc:1238] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:18.726696 31874 rpcz_store.cc:276] 0831 07:59:17.601283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.602012 (+ 729us) service_pool.cc:225] Handling call
0831 07:59:17.604405 (+ 2393us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:17.604721 (+ 316us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 07:59:18.333792 (+729071us) spinlock_profiling.cc:243] Waited 4.85 ms on lock 0x7b1800370600. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce507e0f7 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce5098805 00007fdceb9e2327 00007fdce8007957 00007fdce801706f 00007fdce8016fe8
0831 07:59:18.464336 (+130544us) log.cc:844] Serialized 14 byte log entry
0831 07:59:18.666768 (+202432us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:18.690991 (+ 24223us) log.cc:844] Serialized 26 byte log entry
0831 07:59:18.691149 (+ 158us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:18.691181 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:18.691259 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:18.723336 (+ 32077us) raft_consensus.cc:1683] finished
0831 07:59:18.723448 (+ 112us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:18.723608 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3016,"prepare.queue_time_us":947243,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":1624,"raft.queue_time_us":11581,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":57108,"spinlock_wait_cycles":11155328,"thread_start_us":282684,"threads_started":2,"wal-append.queue_time_us":236958}
I20250831 07:59:18.707052 1649 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:18.693327 1600 raft_consensus.cc:2747] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:18.741775 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:18.742436 31877 raft_consensus.cc:1273] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:18.745451 1678 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:18.745621 1659 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:18.745870 1303 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:18.708830 1261 consensus_queue.cc:237] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.746129 1678 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.746275 1659 consensus_queue.cc:1046] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250831 07:59:18.750254 31881 raft_consensus.cc:1238] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:18.752167 31874 raft_consensus.cc:1273] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 07:59:18.726660 1603 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:18.752207 31732 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:18.758529 31883 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
W20250831 07:59:18.766245 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:18.767999 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 269415us
I20250831 07:59:18.769026 31848 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:18.779220 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1733) took 1171 ms. Trace:
I20250831 07:59:18.779712 31734 rpcz_store.cc:276] 0831 07:59:17.607963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:18.356508 (+748545us) service_pool.cc:225] Handling call
0831 07:59:18.356800 (+ 292us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:18.357151 (+ 351us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:18.357181 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:18.357595 (+ 414us) log.cc:844] Serialized 26 byte log entry
0831 07:59:18.546495 (+188900us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:18.546539 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:18.546608 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:18.778949 (+232341us) raft_consensus.cc:1683] finished
0831 07:59:18.778972 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:18.779146 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":120,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"thread_start_us":188523,"threads_started":1,"wal-append.queue_time_us":278102}
I20250831 07:59:18.726773 1385 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.730908 1447 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:18.778023 1044 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:18.784343 1673 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:18.724215 1439 consensus_queue.cc:237] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.714947 1668 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:18.778177 1603 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:18.999284 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:19.011183 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:18.961654 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 332736us
I20250831 07:59:18.911947 31732 raft_consensus.cc:1273] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:19.015110 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:18.994856 1520 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:19.014004 1454 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.543s user 0.012s sys 0.005s
I20250831 07:59:19.013989 31874 raft_consensus.cc:1238] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
W20250831 07:59:19.010692 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 241780us
W20250831 07:59:18.871434 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:19.032946 31699 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:19.039577 31848 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:19.048898 31848 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:19.033757 1454 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.563s user 0.013s sys 0.006s
I20250831 07:59:18.819449 1678 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:18.894057 1684 log.cc:927] Time spent T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20250831 07:59:18.962913 1669 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:19.058847 31699 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:18.852787 1469 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:18.834861 32464 raft_consensus.cc:1273] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:18.852104 1367 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:19.068994 31698 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:19.062321 31874 raft_consensus.cc:1273] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 07:59:19.038862 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:19.068130 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 45390.4us
I20250831 07:59:19.060631 1624 raft_consensus.cc:2802] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:18.939944 1401 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.124s
I20250831 07:59:19.033378 1673 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.064098 1661 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:19.101251 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:19.105690 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:19.091910 1378 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:19.092113 1678 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:19.115267 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:19.122920 31874 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:19.124013 32434 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:19.092420 1700 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:19.125025 31874 raft_consensus.cc:1238] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
W20250831 07:59:19.149109 1682 log.cc:927] Time spent T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20250831 07:59:19.150373 1628 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.091024 1681 consensus_queue.cc:1035] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20250831 07:59:19.089862 1683 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:19.151520 1261 consensus_queue.cc:1035] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:59:19.152402 1662 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:19.146178 1678 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.152817 1474 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:19.111492 1385 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:19.153367 1668 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:19.152367 1466 consensus_queue.cc:1035] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:59:19.152022 1447 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:19.090027 1669 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:19.155512 1264 consensus_meta.cc:287] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.571s user 0.009s sys 0.001s
I20250831 07:59:19.108507 1624 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:19.178435 31874 raft_consensus.cc:1273] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:19.113483 1661 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:19.192083 31877 raft_consensus.cc:1238] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:19.197564 1695 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:19.198697 1673 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.200007 1624 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:19.211836 31728 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250831 07:59:19.200454 1628 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:19.201012 1547 consensus_meta.cc:287] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.585s user 0.009s sys 0.000s
I20250831 07:59:19.184302 1474 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:19.207109 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:19.210384 31874 raft_consensus.cc:1238] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:19.124215 1458 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.125349 1378 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.203228 1264 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:19.230387 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:19.164037 1700 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.210920 1661 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:19.241998 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 131416us
I20250831 07:59:19.221091 1474 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:19.253680 1264 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:19.254832 1261 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:19.255343 1261 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:19.238766 1695 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:19.257938 1261 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:19.258340 1695 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:19.259025 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:19.259862 31807 raft_consensus.cc:2408] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250831 07:59:19.224139 1547 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:19.270965 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 495219us
I20250831 07:59:19.214471 1662 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.261373 1603 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.238299 1673 consensus_queue.cc:1046] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
W20250831 07:59:19.288364 1508 consensus_meta.cc:287] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.670s user 0.010s sys 0.001s
I20250831 07:59:19.340036 1508 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:19.342059 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:19.342872 31733 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:19.344782 31775 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:19.357257 1705 raft_consensus.cc:2802] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250831 07:59:19.357645 1698 log.cc:927] Time spent T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
I20250831 07:59:19.360641 1458 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.362203 32469 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
W20250831 07:59:19.416616 1710 log.cc:927] Time spent T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
W20250831 07:59:19.439167 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:19.363370 1477 consensus_meta.cc:287] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.831s user 0.004s sys 0.005s
I20250831 07:59:19.297672 1583 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:19.389611 1692 log.cc:927] Time spent T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
W20250831 07:59:19.485123 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:19.486336 31773 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:19.470872 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:19.471587 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:19.357725 1705 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:19.481449 32467 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250831 07:59:19.492682 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 350) took 1861 ms. Trace:
I20250831 07:59:19.493141 32467 rpcz_store.cc:276] 0831 07:59:17.631297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.657338 (+ 26041us) service_pool.cc:225] Handling call
0831 07:59:19.480849 (+1823511us) spinlock_profiling.cc:243] Waited 6.45 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:19.481102 (+ 253us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:19.492309 (+ 11207us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:19.492614 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14843136}
I20250831 07:59:19.491333 31733 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:19.479763 31772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.016s sys 0.013s
W20250831 07:59:19.495096 31772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s user 0.016s sys 0.013s
W20250831 07:59:19.495967 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:19.501909 31733 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:19.504307 1704 raft_consensus.cc:2747] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:19.510301 1704 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:19.511556 1704 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.513033 1665 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:19.509404 1683 raft_consensus.cc:2747] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:19.513880 1665 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.514490 1704 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.517436 1665 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:19.533953 1603 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:19.534646 1603 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:19.537652 1603 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.491990 1583 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:19.540966 1547 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:19.552541 32436 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.022s sys 0.039s
W20250831 07:59:19.570011 32436 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s user 0.022s sys 0.047s
W20250831 07:59:19.568703 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:19.571264 31847 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:19.582809 31775 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:19.585856 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:19.362987 1474 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:19.540665 1581 log.cc:927] Time spent T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
W20250831 07:59:19.564117 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 77878us
I20250831 07:59:19.478408 1673 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:19.479148 1261 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:19.543092 1694 log.cc:927] Time spent T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20250831 07:59:19.541061 1583 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:19.543144 1678 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.541837 1487 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:59:19.635432 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:19.635869 31734 raft_consensus.cc:1273] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:19.638150 1669 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.588404 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:19.596670 1466 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:19.596344 1547 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:19.610644 1367 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:19.610992 1378 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:19.543327 1705 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:19.669629 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39068 (request call id 613) took 1070 ms. Trace:
I20250831 07:59:19.626472 1508 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:19.648411 1649 consensus_meta.cc:287] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.902s user 0.011s sys 0.001s
W20250831 07:59:19.661774 1701 log.cc:927] Time spent T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.244s user 0.001s sys 0.000s
I20250831 07:59:19.661897 1700 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.662201 1583 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:19.658608 32469 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
W20250831 07:59:19.663877 1706 log.cc:927] Time spent T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.170s user 0.000s sys 0.000s
I20250831 07:59:19.660466 1547 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.657311 1466 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:19.666489 1639 log.cc:927] Time spent T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 1.201s user 0.007s sys 0.015s
W20250831 07:59:19.668819 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 68091.4us
W20250831 07:59:19.659869 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 77199us
I20250831 07:59:19.687682 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 253) took 1958 ms. Trace:
I20250831 07:59:19.688104 31809 rpcz_store.cc:276] 0831 07:59:17.728875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.729412 (+ 537us) service_pool.cc:225] Handling call
0831 07:59:17.729758 (+ 346us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:17.730078 (+ 320us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:17.730105 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:17.730582 (+ 477us) log.cc:844] Serialized 26 byte log entry
0831 07:59:18.209820 (+479238us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:18.209881 (+ 61us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:18.209978 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:19.665677 (+1455699us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:19.687400 (+ 21723us) raft_consensus.cc:1683] finished
0831 07:59:19.687432 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:19.687630 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2689,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":64745,"spinlock_wait_cycles":7406336,"thread_start_us":478874,"threads_started":1,"wal-append.queue_time_us":720597}
I20250831 07:59:19.690224 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:19.694833 1705 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.697269 1705 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.697553 31809 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:19.685843 31773 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:19.670061 31802 rpcz_store.cc:276] 0831 07:59:18.598785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:18.621324 (+ 22539us) service_pool.cc:225] Handling call
0831 07:59:19.669053 (+1047729us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b54002f0654. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f81cd0 00007fdce8005eea 00007fdcecf5e60e 00007fdce65fbcc7 00007fdce65fbc52 00007fdce65fbb81 00007fdce65fbafc 00007fdce65fadb2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:19.669569 (+ 516us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42684800}
I20250831 07:59:19.674666 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1536) took 2718 ms. Trace:
I20250831 07:59:19.700387 31801 rpcz_store.cc:276] 0831 07:59:16.956967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:16.977602 (+ 20635us) service_pool.cc:225] Handling call
0831 07:59:17.206748 (+229146us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:17.207450 (+ 702us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:19.674483 (+2467033us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b540009ea50. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:19.674612 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":521858944}
I20250831 07:59:19.615496 1673 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.674940 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:19.650508 1660 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.026s sys 0.113s
I20250831 07:59:19.679178 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1708) took 2664 ms. Trace:
I20250831 07:59:19.717926 31806 rpcz_store.cc:276] 0831 07:59:17.015338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.027296 (+ 11958us) service_pool.cc:225] Handling call
0831 07:59:17.027487 (+ 191us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:17.027623 (+ 136us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 07:59:17.263528 (+235905us) log.cc:844] Serialized 14 byte log entry
0831 07:59:19.676737 (+2413209us) spinlock_profiling.cc:243] Waited 1.12 ms on lock 0x7b18004203a0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce7c4955c 00007fdce507e77b 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdce7c3b2bb 00007fdce7c40dc3 00007fdce7c412da
0831 07:59:19.678827 (+ 2090us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:19.678885 (+ 58us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:19.678935 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:19.679006 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:19.679121 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5924,"prepare.queue_time_us":329885,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":5030,"raft.queue_time_us":1964,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":2013,"spinlock_wait_cycles":2569600,"thread_start_us":314991,"threads_started":2,"wal-append.queue_time_us":196443,"wal-append.run_cpu_time_us":1139,"wal-append.run_wall_time_us":1024283}
I20250831 07:59:19.720156 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:19.624451 1261 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:19.720831 1673 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.721371 1662 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.722163 1261 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:19.639554 1487 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.669682 1458 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:19.660104 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:19.681164 1667 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:19.674461 1704 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:19.672653 1367 raft_consensus.cc:695] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:19.701289 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:19.732905 31802 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:19.734233 32435 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:19.705987 31773 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:19.709316 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:19.735723 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:19.735622 31809 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:19.674142 1624 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.736678 31802 raft_consensus.cc:2391] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:19.721382 1244 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:19.737449 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:19.738132 31848 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:19.717705 1705 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
W20250831 07:59:19.725507 1660 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.784s user 0.026s sys 0.114s
I20250831 07:59:19.740207 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:19.741411 31809 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:19.742182 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:19.813957 31802 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:19.814015 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:19.743592 1269 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:19.743221 1244 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:19.811532 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:19.808972 1378 raft_consensus.cc:2802] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:19.820624 31801 raft_consensus.cc:2391] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:19.751303 31847 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:19.814800 31809 raft_consensus.cc:2391] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:59:19.814965 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 67210.6us
I20250831 07:59:19.814915 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:19.815680 32435 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
W20250831 07:59:19.822153 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 70953.1us
I20250831 07:59:19.822758 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:19.823936 31809 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:19.811314 1264 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:19.825364 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:19.807924 1626 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:19.737852 1667 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.817839 1487 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:19.743505 1458 raft_consensus.cc:695] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:19.814211 1706 log.cc:927] Time spent T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20250831 07:59:19.815903 31848 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:19.808089 1474 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.755285 1712 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:19.808610 1705 raft_consensus.cc:695] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:19.740875 1704 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:19.819173 1269 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:19.747478 1661 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.839524 1269 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:19.840576 1704 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:19.840816 1661 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:19.841867 1705 consensus_queue.cc:237] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.831996 1378 raft_consensus.cc:695] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:19.821215 1703 consensus_queue.cc:1035] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20250831 07:59:19.833256 1626 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.831861 1264 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:19.838541 1547 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:19.825754 1711 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:19.825224 1385 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:19.844583 1378 consensus_queue.cc:237] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.837482 1466 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.827023 1261 consensus_queue.cc:1046] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20250831 07:59:19.852813 1711 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:19.896842 1711 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:19.894259 1458 consensus_queue.cc:237] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.910311 1624 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:19.911645 1627 raft_consensus.cc:3053] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:19.911798 1626 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:19.912353 1627 raft_consensus.cc:738] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:19.916744 1627 consensus_queue.cc:260] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:19.918427 1627 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:19.919600 1597 log.cc:927] Time spent T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 2.544s user 0.004s sys 0.017s
I20250831 07:59:19.921324 1711 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:19.922669 1667 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.923534 1711 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.825937 31801 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:19.983525 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 160771us
I20250831 07:59:19.824687 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:19.985142 31802 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 8.
I20250831 07:59:19.986222 31773 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:19.987428 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:19.991598 31806 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:19.993398 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 264) took 1179 ms. Trace:
I20250831 07:59:19.993917 1261 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:19.995245 1261 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:19.855935 1244 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:19.832470 31809 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:19.851163 1269 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.004928 1269 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:20.005888 1269 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:20.006842 1269 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:20.008250 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 192297us
W20250831 07:59:20.008630 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:20.012455 1385 raft_consensus.cc:2747] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:19.837186 1367 consensus_queue.cc:237] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.001299 1264 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:19.988988 31802 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:20.014007 1244 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:19.994038 31807 rpcz_store.cc:276] 0831 07:59:18.813659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:19.271625 (+457966us) service_pool.cc:225] Handling call
0831 07:59:19.272568 (+ 943us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:19.274203 (+ 1635us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:19.274247 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:19.278187 (+ 3940us) log.cc:844] Serialized 26 byte log entry
0831 07:59:19.659767 (+381580us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:19.659833 (+ 66us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:19.659907 (+ 74us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:19.992720 (+332813us) raft_consensus.cc:1683] finished
0831 07:59:19.992774 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:19.993116 (+ 342us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1168,"raft.queue_time_us":1886,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":10792,"thread_start_us":381240,"threads_started":1,"wal-append.queue_time_us":528008}
I20250831 07:59:20.022264 31809 raft_consensus.cc:1273] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 07:59:20.026115 1269 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:20.028718 32435 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:20.029098 31848 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:19.999191 1261 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:59:20.038942 1547 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:20.039671 1547 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:20.040277 1547 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:20.041698 1626 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:20.042758 1626 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.044286 1628 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:20.045266 1628 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.045884 1650 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:20.046902 1650 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.052387 1667 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:20.078091 1667 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:20.078676 1667 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:20.053506 1705 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:20.080355 1705 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:20.083335 1705 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:20.090054 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 61603us
I20250831 07:59:20.053440 1700 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:20.053371 1672 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:20.091022 1700 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:20.091271 1672 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:20.093086 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:20.089612 1547 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:20.083456 31800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1572) took 2519 ms (client timeout 2132 ms). Trace:
I20250831 07:59:20.054069 1520 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20250831 07:59:20.096774 31800 rpcz_store.cc:269] 0831 07:59:17.563774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.570944 (+ 7170us) service_pool.cc:225] Handling call
0831 07:59:20.082911 (+2511967us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b540009ea54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:20.083228 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":954414464}
I20250831 07:59:20.093479 1700 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:20.112124 32465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 342) took 3741 ms. Trace:
I20250831 07:59:20.113366 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1722) took 2424 ms. Trace:
I20250831 07:59:20.114840 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1243) took 2521 ms. Trace:
I20250831 07:59:20.089682 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:20.115979 31801 raft_consensus.cc:2408] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250831 07:59:20.116750 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1699) took 4645 ms. Trace:
I20250831 07:59:20.116868 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:20.093521 1264 raft_consensus.cc:2802] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:20.100798 1520 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:20.110581 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:20.100438 1662 consensus_queue.cc:1035] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.448s
W20250831 07:59:20.219534 1032 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20250831 07:59:20.500101 32466 raft_consensus.cc:2391] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:20.153416 1547 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:20.500643 31807 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:20.121188 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:20.118642 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:20.502352 31809 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:20.502887 31800 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:20.503221 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:20.331239 1505 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:20.504073 31801 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:20.270008 940 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:20.448719 1626 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:20.450168 1628 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:20.396426 1650 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:20.108115 1672 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:20.117647 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:20.492471 1495 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:20.471252 1240 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:20.507390 31802 raft_consensus.cc:2408] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20250831 07:59:20.119277 1588 log.cc:927] Time spent T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 3.096s user 0.019s sys 0.019s
I20250831 07:59:20.119735 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:20.117936 32463 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47886 (request call id 406) took 2484 ms. Trace:
I20250831 07:59:20.471810 1666 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:59:20.118911 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:20.501197 1547 consensus_queue.cc:1035] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 07:59:20.505364 940 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:20.492550 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 194282us
I20250831 07:59:20.509526 31876 rpcz_store.cc:276] 0831 07:59:17.593121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.594850 (+ 1729us) service_pool.cc:225] Handling call
0831 07:59:17.595127 (+ 277us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:17.606369 (+ 11242us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.114196 (+2507827us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b540007cbd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:20.114593 (+ 397us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23836544}
I20250831 07:59:20.502004 31726 rpcz_store.cc:276] 0831 07:59:15.471354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:15.556498 (+ 85144us) service_pool.cc:225] Handling call
0831 07:59:15.583184 (+ 26686us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:18.147368 (+2564184us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.078665 (+1931297us) spinlock_profiling.cc:243] Waited 2.56 s on lock 0x7b54001a9d54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:20.116479 (+ 37814us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5893551360}
I20250831 07:59:20.493517 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:20.537348 32462 raft_consensus.cc:2366] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5.
I20250831 07:59:20.537569 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:20.537860 32462 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47886 (request call id 411) took 1936 ms (client timeout 1998 ms). Trace:
I20250831 07:59:20.538203 1672 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:20.538349 32462 rpcz_store.cc:269] 0831 07:59:18.601890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:18.606855 (+ 4965us) service_pool.cc:225] Handling call
0831 07:59:20.231450 (+1624595us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:20.537785 (+306335us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5400448}
I20250831 07:59:20.538357 31802 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:20.510918 1520 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:20.508332 32467 raft_consensus.cc:2366] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6.
I20250831 07:59:20.510316 31803 rpcz_store.cc:276] 0831 07:59:17.689277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.689427 (+ 150us) service_pool.cc:225] Handling call
0831 07:59:17.689692 (+ 265us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:18.012425 (+322733us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:18.012458 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:18.012508 (+ 50us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:18.012558 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.112860 (+2100302us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b54000cf3d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:20.112955 (+ 95us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.113188 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":741683072}
W20250831 07:59:20.391746 1724 log.cc:927] Time spent T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
W20250831 07:59:20.469658 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:20.536247 940 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:20.533766 32463 rpcz_store.cc:276] 0831 07:59:17.633051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.648453 (+ 15402us) service_pool.cc:225] Handling call
0831 07:59:17.695293 (+ 46840us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:18.130363 (+435070us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:18.131298 (+ 935us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:18.133985 (+ 2687us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:18.136067 (+ 2082us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.117259 (+1981192us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b5400500fd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:20.117359 (+ 100us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.117669 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":952909184}
I20250831 07:59:20.388492 1261 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:20.541704 31801 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:20.122571 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1666) took 1784 ms. Trace:
I20250831 07:59:20.512174 1240 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:20.471483 1668 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:20.499251 32465 rpcz_store.cc:276] 0831 07:59:16.370685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:16.621706 (+251021us) service_pool.cc:225] Handling call
0831 07:59:16.622142 (+ 436us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:17.152594 (+530452us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.111466 (+2958872us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b5400550354. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:20.111872 (+ 406us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1218163584}
I20250831 07:59:20.552345 31875 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:20.515084 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:20.220655 1705 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:20.539083 1672 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:20.560228 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:20.555178 31883 rpcz_store.cc:276] 0831 07:59:18.338231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:18.562280 (+224049us) service_pool.cc:225] Handling call
0831 07:59:20.122067 (+1559787us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b5400247a54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:20.122327 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":427629312}
I20250831 07:59:20.559037 1240 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:20.559397 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.508404 1264 raft_consensus.cc:695] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:20.561726 31809 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:20.566013 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:20.570505 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:20.571170 31809 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 1.
I20250831 07:59:20.568447 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:20.575778 32435 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:20.562436 1667 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.579887 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 212) took 1346 ms. Trace:
I20250831 07:59:20.574509 32464 raft_consensus.cc:2391] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 7: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
W20250831 07:59:20.584487 31876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1258) took 1307 ms (client timeout 1716 ms). Trace:
W20250831 07:59:20.585011 31876 rpcz_store.cc:269] 0831 07:59:19.274014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.557726 (+1283712us) service_pool.cc:225] Handling call
0831 07:59:20.581007 (+ 23281us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:20.569653 1261 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:20.601042 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:20.579437 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:20.602257 31800 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:20.604203 31803 raft_consensus.cc:2391] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:20.574208 1705 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:20.580263 31874 rpcz_store.cc:276] 0831 07:59:19.209169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:19.237274 (+ 28105us) service_pool.cc:225] Handling call
0831 07:59:19.237590 (+ 316us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:19.240904 (+ 3314us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:19.240949 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:19.241485 (+ 536us) log.cc:844] Serialized 26 byte log entry
0831 07:59:19.254645 (+ 13160us) spinlock_profiling.cc:243] Waited 6.35 ms on lock 0x7b48001401d8. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcecebfc41 00007fdceceb6de7 00007fdceceb6c7a 00007fdcecfac06f 00007fdcecfabd37 00007fdcecfc035c 00007fdcecfc0308 00007fdcecfc0288 00007fdcecfc0243 00007fdcecfbf4ec 00007fdcebf3d656
0831 07:59:19.836866 (+582221us) spinlock_profiling.cc:243] Waited 8.73 ms on lock 0x7b18004303c0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce507e0f7 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdce7c3b2bb 00007fdce7c40dc3 00007fdce7c41171 00007fdce7fd009f
0831 07:59:20.296215 (+459349us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:20.313665 (+ 17450us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.355252 (+ 41587us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:20.551632 (+196380us) raft_consensus.cc:1683] finished
0831 07:59:20.551660 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.555268 (+ 3608us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11370,"raft.queue_time_us":2983,"raft.run_cpu_time_us":835,"raft.run_wall_time_us":19425,"spinlock_wait_cycles":34679808,"thread_start_us":197265,"threads_started":1,"wal-append.queue_time_us":1027904}
I20250831 07:59:20.605759 31801 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:20.608592 31802 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:20.591905 1668 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:20.610852 1672 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:20.573451 1666 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.611877 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.612877 31874 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:20.613517 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 214) took 1075 ms. Trace:
I20250831 07:59:20.613659 1666 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:20.613863 31874 rpcz_store.cc:276] 0831 07:59:19.538478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.611505 (+1073027us) service_pool.cc:225] Handling call
0831 07:59:20.613451 (+ 1946us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:20.615187 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 105192us
W20250831 07:59:20.615126 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 118509us
I20250831 07:59:20.591063 32464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38362 (request call id 355) took 1130 ms. Trace:
I20250831 07:59:20.617404 1672 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:20.617503 32464 rpcz_store.cc:276] 0831 07:59:19.459444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:19.478055 (+ 18611us) service_pool.cc:225] Handling call
0831 07:59:20.119439 (+641384us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e5ce 000000000035a3c9 00007fdcecee5300 00007fdcecfabec4 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:20.589644 (+470205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48776448}
I20250831 07:59:20.618281 1672 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.621213 1672 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:20.624253 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:20.625072 32464 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 1.
I20250831 07:59:20.625777 1269 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:20.611562 1668 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:20.626288 1269 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:20.626812 1269 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:20.627100 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1247) took 3018 ms. Trace:
I20250831 07:59:20.627761 1316 raft_consensus.cc:2747] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:20.627501 31878 rpcz_store.cc:276] 0831 07:59:17.608123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.610115 (+ 1992us) service_pool.cc:225] Handling call
0831 07:59:17.610519 (+ 404us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:17.622954 (+ 12435us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.626524 (+3003570us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b54003eec54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:20.626929 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26954624}
I20250831 07:59:20.629047 1316 raft_consensus.cc:2747] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 07:59:20.630300 31882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1236) took 3633 ms (client timeout 2554 ms). Trace:
W20250831 07:59:20.630743 31882 rpcz_store.cc:269] 0831 07:59:16.996533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:16.996767 (+ 234us) service_pool.cc:225] Handling call
0831 07:59:20.629839 (+3633072us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b5400058054. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:20.630091 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1124363008}
I20250831 07:59:20.632117 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:20.632725 31882 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:20.633373 1261 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:20.626304 1705 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:20.629015 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:20.637820 1264 consensus_queue.cc:237] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.642506 31807 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:20.634917 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.644776 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.644757 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:20.645638 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:20.645375 31876 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:20.646840 31878 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 5
I20250831 07:59:20.651327 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1263) took 1188 ms. Trace:
I20250831 07:59:20.651850 31876 rpcz_store.cc:276] 0831 07:59:19.458805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.603345 (+1144540us) service_pool.cc:225] Handling call
0831 07:59:20.647385 (+ 44040us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:20.667346 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:20.680229 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1740) took 1157 ms. Trace:
I20250831 07:59:20.682788 31882 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:20.683008 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 265) took 1183 ms. Trace:
I20250831 07:59:20.680850 31801 rpcz_store.cc:276] 0831 07:59:19.499105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.616384 (+1117279us) service_pool.cc:225] Handling call
0831 07:59:20.642084 (+ 25700us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:20.648974 (+ 6890us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:20.649015 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:20.655934 (+ 6919us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:20.655995 (+ 61us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.656077 (+ 82us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.656227 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:20.683327 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 215) took 1037 ms. Trace:
I20250831 07:59:20.683673 31882 rpcz_store.cc:276] 0831 07:59:19.645947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.631645 (+985698us) service_pool.cc:225] Handling call
0831 07:59:20.683275 (+ 51630us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:20.684621 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.684779 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1744) took 1030 ms. Trace:
I20250831 07:59:20.685585 31882 raft_consensus.cc:2408] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:20.686038 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1261) took 1228 ms. Trace:
I20250831 07:59:20.687054 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.729835 31773 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:20.700942 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.791684 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1743) took 1158 ms. Trace:
I20250831 07:59:20.791756 31874 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:20.686224 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1265) took 1566 ms. Trace:
I20250831 07:59:20.686905 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 213) took 1430 ms. Trace:
I20250831 07:59:20.792162 31800 rpcz_store.cc:276] 0831 07:59:19.624914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.694793 (+1069879us) service_pool.cc:225] Handling call
0831 07:59:20.715664 (+ 20871us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:20.770299 (+ 54635us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:20.770346 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:20.776113 (+ 5767us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:20.783124 (+ 7011us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.783200 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.783317 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:20.727258 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:20.685122 31801 rpcz_store.cc:276] 0831 07:59:19.653968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.683835 (+1029867us) service_pool.cc:225] Handling call
0831 07:59:20.684200 (+ 365us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:20.684450 (+ 250us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:20.684472 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:20.684521 (+ 49us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:20.684558 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.684621 (+ 63us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.684714 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:20.793059 31883 rpcz_store.cc:276] 0831 07:59:19.250160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.634596 (+1384436us) service_pool.cc:225] Handling call
0831 07:59:20.680105 (+ 45509us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:20.703639 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 61515.1us
W20250831 07:59:20.795449 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 65715.4us
I20250831 07:59:20.782085 31882 rpcz_store.cc:276] 0831 07:59:19.457291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.684289 (+1226998us) service_pool.cc:225] Handling call
0831 07:59:20.685982 (+ 1693us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:20.683336 31802 rpcz_store.cc:276] 0831 07:59:19.499343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.636616 (+1137273us) service_pool.cc:225] Handling call
0831 07:59:20.652893 (+ 16277us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:20.673229 (+ 20336us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:20.673259 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:20.682746 (+ 9487us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:20.682786 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.682845 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.682959 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:20.797224 32435 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:20.798142 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1262) took 1335 ms. Trace:
I20250831 07:59:20.798632 31874 rpcz_store.cc:276] 0831 07:59:19.457958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.634964 (+1177006us) service_pool.cc:225] Handling call
0831 07:59:20.792867 (+157903us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:20.771641 1654 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.968s user 0.006s sys 0.035s
I20250831 07:59:20.722913 1577 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:20.695457 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:20.715404 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:20.801690 32435 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:20.805841 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.727602 1498 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:20.760092 1667 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:20.814981 1577 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.815956 1018 raft_consensus.cc:2802] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250831 07:59:20.802623 1654 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.138s user 0.007s sys 0.042s
I20250831 07:59:20.727504 1585 raft_consensus.cc:3053] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:20.792984 31729 rpcz_store.cc:276] 0831 07:59:19.120001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:19.120430 (+ 429us) service_pool.cc:225] Handling call
0831 07:59:19.120725 (+ 295us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:19.120909 (+ 184us) raft_consensus.cc:1532] Early marking committed up to index 5
0831 07:59:19.145109 (+ 24200us) log.cc:844] Serialized 14 byte log entry
0831 07:59:20.666516 (+1521407us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:20.685716 (+ 19200us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:20.685791 (+ 75us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.685882 (+ 91us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.686036 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1481634,"prepare.queue_time_us":24325,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":1492440,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":40158,"threads_started":1}
I20250831 07:59:20.724563 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:20.723780 1269 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.823930 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.828299 31876 raft_consensus.cc:2408] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 07:59:20.830905 32435 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:20.805059 1662 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:20.840885 31734 raft_consensus.cc:2408] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:20.834148 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:20.729331 1261 consensus_queue.cc:1046] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.100s
I20250831 07:59:20.822191 1018 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:20.843683 32468 raft_consensus.cc:2391] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:20.844410 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:20.825364 1585 raft_consensus.cc:738] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:20.845990 31883 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:20.848084 1438 raft_consensus.cc:2747] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:20.848474 1538 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 3.690s user 0.035s sys 0.022s
I20250831 07:59:20.854951 1269 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:20.844424 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1747) took 1159 ms. Trace:
I20250831 07:59:20.850653 1474 raft_consensus.cc:2802] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:20.850862 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 273) took 1027 ms. Trace:
I20250831 07:59:20.852150 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.854197 1606 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:20.874008 31802 rpcz_store.cc:276] 0831 07:59:19.685494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.799658 (+1114164us) service_pool.cc:225] Handling call
0831 07:59:20.829301 (+ 29643us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:20.842089 (+ 12788us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:20.842116 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:20.844163 (+ 2047us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:20.844202 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.844260 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.844363 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:20.876142 31874 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:20.877131 31800 rpcz_store.cc:276] 0831 07:59:19.823096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.820294 (+997198us) service_pool.cc:225] Handling call
0831 07:59:20.850224 (+ 29930us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:20.850528 (+ 304us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 07:59:20.850551 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:20.850598 (+ 47us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:20.850639 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.850696 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.850802 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:20.878449 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1256) took 2162 ms. Trace:
I20250831 07:59:20.878741 31876 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
W20250831 07:59:20.854983 1538 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 3.748s user 0.035s sys 0.023s
I20250831 07:59:20.884622 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:20.887753 31878 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:20.847216 1018 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:59:20.893486 1525 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:20.898152 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.898476 31880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1647) took 2816 ms. Trace:
I20250831 07:59:20.898908 31880 rpcz_store.cc:276] 0831 07:59:18.081824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:18.081954 (+ 130us) service_pool.cc:225] Handling call
0831 07:59:20.888191 (+2806237us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b54002bbe54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:20.898298 (+ 10107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1348784896}
I20250831 07:59:20.900043 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.901223 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.851255 1731 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:20.904044 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.884506 1711 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:20.906980 1606 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.879760 1662 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.911799 32470 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:20.912304 1585 consensus_queue.cc:260] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.912997 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1240) took 3320 ms. Trace:
I20250831 07:59:20.911870 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1272) took 1642 ms. Trace:
I20250831 07:59:20.914170 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:20.907476 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1258) took 2151 ms. Trace:
I20250831 07:59:20.943665 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:20.940812 31726 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:20.933535 31883 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:20.943660 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:20.960716 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.961067 31880 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:20.961285 1668 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:20.961565 31883 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:20.962030 1668 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:20.962543 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.963523 31883 raft_consensus.cc:2408] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:20.964007 31848 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:20.964300 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:20.964395 1668 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:20.964944 1547 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:20.965092 31883 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:20.946172 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.916440 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1746) took 1234 ms. Trace:
I20250831 07:59:20.902173 1474 raft_consensus.cc:695] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:20.936837 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1722) took 3297 ms. Trace:
I20250831 07:59:20.968206 31801 rpcz_store.cc:276] 0831 07:59:19.682054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.794206 (+1112152us) service_pool.cc:225] Handling call
0831 07:59:20.830707 (+ 36501us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:20.916114 (+ 85407us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:20.916143 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:20.916190 (+ 47us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:20.916224 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.916284 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.916381 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:20.965641 1547 raft_consensus.cc:695] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:20.976279 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.913792 1585 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:20.913898 1508 raft_consensus.cc:2802] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 07:59:20.978327 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:20.979324 31876 raft_consensus.cc:2391] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:20.976917 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.980307 1662 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:20.980746 1653 log.cc:927] Time spent T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 2.420s user 0.005s sys 0.000s
I20250831 07:59:20.981083 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1269) took 1750 ms. Trace:
I20250831 07:59:20.896405 1498 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:20.992342 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 1.93333e+06us
I20250831 07:59:20.991382 31725 rpcz_store.cc:276] 0831 07:59:19.231069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:19.231242 (+ 173us) service_pool.cc:225] Handling call
0831 07:59:19.231569 (+ 327us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:19.231774 (+ 205us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 07:59:19.265830 (+ 34056us) log.cc:844] Serialized 14 byte log entry
0831 07:59:20.814219 (+1548389us) log.cc:844] Serialized 14 byte log entry
0831 07:59:20.967770 (+153551us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:20.973840 (+ 6070us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:20.977414 (+ 3574us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.980719 (+ 3305us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.980898 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1414078,"prepare.queue_time_us":162262,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":85,"raft.queue_time_us":1417093,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":18345,"thread_start_us":244974,"threads_started":3}
I20250831 07:59:20.993580 1498 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:20.993690 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:20.994233 31699 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:20.926926 1275 raft_consensus.cc:3053] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:20.995008 1275 raft_consensus.cc:738] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:59:20.995232 1599 log.cc:927] Time spent T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 3.622s user 0.000s sys 0.009s
W20250831 07:59:20.981290 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 1.84112e+06us
I20250831 07:59:20.996726 31698 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:20.997012 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.997558 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1238) took 3995 ms. Trace:
I20250831 07:59:20.997941 31875 rpcz_store.cc:276] 0831 07:59:17.002031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:19.239912 (+2237881us) service_pool.cc:225] Handling call
0831 07:59:20.551920 (+1312008us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:20.563806 (+ 11886us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.997340 (+433534us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b5400298bd0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:20.997497 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2987183360}
I20250831 07:59:20.998431 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.998715 31698 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:20.999220 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.002627 1648 consensus_queue.cc:1035] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:59:21.002995 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 266) took 1503 ms. Trace:
I20250831 07:59:20.983960 1711 raft_consensus.cc:695] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:20.986657 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:20.927755 1731 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:20.937268 1745 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:20.992655 31808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1715) took 3778 ms. Trace:
I20250831 07:59:20.954279 31732 rpcz_store.cc:276] 0831 07:59:19.269607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:19.269797 (+ 190us) service_pool.cc:225] Handling call
0831 07:59:19.294158 (+ 24361us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:19.294367 (+ 209us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 07:59:19.305699 (+ 11332us) log.cc:844] Serialized 14 byte log entry
0831 07:59:20.772391 (+1466692us) log.cc:844] Serialized 14 byte log entry
0831 07:59:20.904746 (+132355us) log.cc:844] Serialized 14 byte log entry
0831 07:59:20.911303 (+ 6557us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:20.911384 (+ 81us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:20.911458 (+ 74us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.911546 (+ 88us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.911679 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1392125,"prepare.queue_time_us":159663,"prepare.run_cpu_time_us":139,"prepare.run_wall_time_us":25344,"raft.queue_time_us":1389528,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":4920,"thread_start_us":179287,"threads_started":3,"wal-append.queue_time_us":62282}
I20250831 07:59:21.000061 1525 raft_consensus.cc:2747] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:21.000679 1679 raft_consensus.cc:2747] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:20.989959 1508 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:21.011850 1508 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:59:21.000742 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:20.994663 31725 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:21.000212 31880 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
W20250831 07:59:21.001818 31806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1731) took 1504 ms (client timeout 1924 ms). Trace:
I20250831 07:59:20.945322 1520 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.004739 31848 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:21.014771 31734 raft_consensus.cc:2408] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:20.985425 1474 consensus_queue.cc:237] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.003441 31809 rpcz_store.cc:276] 0831 07:59:19.499560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.636485 (+1136925us) service_pool.cc:225] Handling call
0831 07:59:21.001788 (+365303us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:21.002013 (+ 225us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:21.002036 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:21.002076 (+ 40us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:21.002107 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:21.002161 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:21.002785 (+ 624us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b540009f450. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:21.002915 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":756744576}
I20250831 07:59:21.016969 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:21.005326 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 94426.3us
I20250831 07:59:21.006343 31882 raft_consensus.cc:2408] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:21.005733 31729 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:21.010635 31877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1667) took 2650 ms. Trace:
W20250831 07:59:21.012880 31806 rpcz_store.cc:269] 0831 07:59:19.496959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:19.719899 (+222940us) service_pool.cc:225] Handling call
0831 07:59:21.001628 (+1281729us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b540009f454. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:21.001743 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":622237952}
I20250831 07:59:21.018646 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.001528 1275 consensus_queue.cc:260] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.013679 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:21.014577 31876 raft_consensus.cc:2391] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:20.913400 31879 rpcz_store.cc:276] 0831 07:59:17.592290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.593936 (+ 1646us) service_pool.cc:225] Handling call
0831 07:59:17.601844 (+ 7908us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:17.603064 (+ 1220us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:17.603091 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:17.607534 (+ 4443us) log.cc:844] Serialized 26 byte log entry
0831 07:59:20.630845 (+3023311us) spinlock_profiling.cc:243] Waited 4.54 ms on lock 0x7b1800296260. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce507e0f7 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdce7c3b2bb 00007fdce7c40dc3 00007fdce7c41171 00007fdce7fd009f
0831 07:59:20.804458 (+173613us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:20.804522 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.818719 (+ 14197us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:20.912591 (+ 93872us) raft_consensus.cc:1683] finished
0831 07:59:20.912611 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.912827 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8300,"raft.queue_time_us":1822,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":10903,"spinlock_wait_cycles":10449664,"thread_start_us":139883,"threads_started":1,"wal-append.queue_time_us":3206826}
I20250831 07:59:20.971293 31730 rpcz_store.cc:276] 0831 07:59:17.528763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.528955 (+ 192us) service_pool.cc:225] Handling call
0831 07:59:17.529389 (+ 434us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:17.549259 (+ 19870us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:17.549292 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:17.562544 (+ 13252us) log.cc:844] Serialized 26 byte log entry
0831 07:59:19.723060 (+2160516us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b18001b96a0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce507e0f7 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdce7c3b2bb 00007fdce7c40dc3 00007fdce7c41171 00007fdce7fd009f
0831 07:59:19.962878 (+239818us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:19.963754 (+ 876us) log.cc:844] Serialized 14 byte log entry
0831 07:59:20.783989 (+820235us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.799479 (+ 15490us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:20.804052 (+ 4573us) raft_consensus.cc:1683] finished
0831 07:59:20.812775 (+ 8723us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.825661 (+ 12886us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":667599,"prepare.queue_time_us":70252,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":664877,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":31582,"spinlock_wait_cycles":53864576,"thread_start_us":398063,"threads_started":2,"wal-append.queue_time_us":2331126}
I20250831 07:59:21.024816 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.025513 31878 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:21.027330 31874 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:21.028239 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.018170 31877 rpcz_store.cc:276] 0831 07:59:18.338515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:18.744708 (+406193us) service_pool.cc:225] Handling call
0831 07:59:18.749882 (+ 5174us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:19.204693 (+454811us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.949461 (+1744768us) spinlock_profiling.cc:243] Waited 442 ms on lock 0x7b5400247054. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:20.988610 (+ 39149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1016445824}
I20250831 07:59:21.017344 1498 raft_consensus.cc:2747] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:21.037143 31880 raft_consensus.cc:2366] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7.
I20250831 07:59:21.033408 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.034073 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:21.038215 31809 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 8
I20250831 07:59:20.955391 31731 rpcz_store.cc:276] 0831 07:59:18.640392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:18.641952 (+ 1560us) service_pool.cc:225] Handling call
0831 07:59:18.642326 (+ 374us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:18.642552 (+ 226us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 07:59:18.642867 (+ 315us) log.cc:844] Serialized 14 byte log entry
0831 07:59:19.055077 (+412210us) log.cc:844] Serialized 14 byte log entry
0831 07:59:20.611212 (+1556135us) spinlock_profiling.cc:243] Waited 5.13 ms on lock 0x7b180034fa20. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce501365a 00007fdcee4e4409 00007fdce4ed4af0 00007fdce5098894 00007fdce5098805 00007fdce7ff5020 00007fdce80076c6 00007fdce8018237 00007fdce80181b8
0831 07:59:20.759935 (+148723us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:20.770371 (+ 10436us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:20.772607 (+ 2236us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.783038 (+ 10431us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.791476 (+ 8438us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1623878,"prepare.queue_time_us":20130,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":84,"raft.queue_time_us":1625937,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":19425,"spinlock_wait_cycles":11795200,"thread_start_us":277673,"threads_started":2,"wal-append.queue_time_us":513472,"wal-append.run_cpu_time_us":1809,"wal-append.run_wall_time_us":1075742}
W20250831 07:59:21.035734 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 68971.4us
I20250831 07:59:21.036365 31729 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:21.008838 31808 rpcz_store.cc:276] 0831 07:59:17.214411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.224760 (+ 10349us) service_pool.cc:225] Handling call
0831 07:59:17.225110 (+ 350us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:17.228532 (+ 3422us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:17.228574 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:17.229114 (+ 540us) log.cc:844] Serialized 26 byte log entry
0831 07:59:18.230988 (+1001874us) spinlock_profiling.cc:243] Waited 993 us on lock 0x7b1800420340. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce507e0f7 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdce7c3b2bb 00007fdce7c40dc3 00007fdce7c41171 00007fdce7fd009f
0831 07:59:18.438286 (+207298us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:18.438651 (+ 365us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:18.439048 (+ 397us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:20.830482 (+2391434us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:20.992301 (+161819us) raft_consensus.cc:1683] finished
0831 07:59:20.992321 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.992494 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":10520,"spinlock_wait_cycles":27978240,"thread_start_us":205151,"threads_started":1,"wal-append.queue_time_us":1319805}
I20250831 07:59:21.040854 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 267) took 1541 ms. Trace:
I20250831 07:59:21.040277 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 218) took 1019 ms. Trace:
I20250831 07:59:21.041836 31875 rpcz_store.cc:276] 0831 07:59:20.021548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.013174 (+991626us) service_pool.cc:225] Handling call
0831 07:59:21.040219 (+ 27045us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:21.040155 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:21.051604 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:21.043226 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.044253 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.053623 31848 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:21.053980 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.041520 31807 rpcz_store.cc:276] 0831 07:59:19.499850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.644606 (+1144756us) service_pool.cc:225] Handling call
0831 07:59:20.687279 (+ 42673us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:20.723624 (+ 36345us) raft_consensus.cc:1532] Early marking committed up to index 6
0831 07:59:20.844380 (+120756us) log.cc:844] Serialized 14 byte log entry
0831 07:59:21.040434 (+196054us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:21.040506 (+ 72us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:21.040569 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:21.040671 (+ 102us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:21.040801 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":287245,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":4084,"raft.queue_time_us":168,"thread_start_us":147546,"threads_started":2}
I20250831 07:59:21.054468 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:21.054827 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.069608 31698 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:21.053725 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.053645 31732 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:21.042989 1275 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:21.072440 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.070434 31773 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:21.052244 1668 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:21.075605 31876 raft_consensus.cc:2391] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:21.069422 31725 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:21.075169 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.041285 1672 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:21.073845 31730 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
W20250831 07:59:21.078302 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 117989us
I20250831 07:59:21.080482 32434 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:21.083266 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1270) took 1847 ms. Trace:
I20250831 07:59:21.089183 31883 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:21.092504 31729 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:21.095548 1520 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:21.095803 1666 raft_consensus.cc:2762] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20250831 07:59:21.099213 31875 raft_consensus.cc:1238] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:21.100121 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1259) took 1825 ms. Trace:
I20250831 07:59:21.100548 31875 rpcz_store.cc:276] 0831 07:59:19.274410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.042783 (+1768373us) service_pool.cc:225] Handling call
0831 07:59:21.067286 (+ 24503us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:21.099714 (+ 32428us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:21.100034 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:21.102100 1625 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:21.102542 1018 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.103811 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.102849 1625 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.104471 31806 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:21.105587 31725 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250831 07:59:21.106264 1625 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:21.103111 31731 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:21.103952 31800 raft_consensus.cc:1238] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
W20250831 07:59:21.112396 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:21.112854 31731 raft_consensus.cc:1273] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20250831 07:59:21.113426 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:21.114159 31731 raft_consensus.cc:3053] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:21.114614 31731 raft_consensus.cc:738] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:21.117055 31731 consensus_queue.cc:260] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.114229 1648 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:21.118567 1731 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:21.121502 31726 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:21.120232 31876 raft_consensus.cc:1273] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 07:59:21.052879 1745 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.088292 1711 consensus_queue.cc:237] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.124619 31731 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:21.122732 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 90076us
I20250831 07:59:21.123302 1032 raft_consensus.cc:2747] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:21.126796 1498 raft_consensus.cc:2747] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:21.126741 31698 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:21.127111 1745 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:21.129036 31698 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:21.129258 31878 raft_consensus.cc:1273] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250831 07:59:21.129910 1525 consensus_queue.cc:1046] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:59:21.130532 1679 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 07:59:21.121628 31734 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:21.120692 31848 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:21.130125 31698 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:21.133828 1523 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:21.134420 1679 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:21.135910 31698 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:21.129410 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 216) took 1473 ms. Trace:
I20250831 07:59:21.075963 1547 consensus_queue.cc:237] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.137233 31879 rpcz_store.cc:276] 0831 07:59:19.656546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.076662 (+1420116us) service_pool.cc:225] Handling call
0831 07:59:21.128735 (+ 52073us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:21.129083 (+ 348us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:21.129107 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:21.129181 (+ 74us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:21.129218 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:21.129271 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:21.129361 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:21.137995 31877 raft_consensus.cc:2408] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 07:59:21.125708 31726 raft_consensus.cc:3053] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:21.138976 31726 raft_consensus.cc:738] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:21.132557 31807 raft_consensus.cc:1273] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:59:21.083746 31728 rpcz_store.cc:276] 0831 07:59:19.235377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:19.235535 (+ 158us) service_pool.cc:225] Handling call
0831 07:59:19.235969 (+ 434us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:20.781142 (+1545173us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:20.781258 (+ 116us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:20.839929 (+ 58671us) log.cc:844] Serialized 26 byte log entry
0831 07:59:20.968835 (+128906us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:20.968868 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.989554 (+ 20686us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:21.082913 (+ 93359us) raft_consensus.cc:1683] finished
0831 07:59:21.082941 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:21.083066 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1389336,"raft.queue_time_us":1401236,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":137,"thread_start_us":201820,"threads_started":2,"wal-append.queue_time_us":229634}
I20250831 07:59:21.140074 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1267) took 1055 ms. Trace:
I20250831 07:59:21.141930 31876 rpcz_store.cc:276] 0831 07:59:20.084653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.084826 (+1000173us) service_pool.cc:225] Handling call
0831 07:59:21.087256 (+ 2430us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:21.122210 (+ 34954us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:21.139995 (+ 17785us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:21.128017 1745 raft_consensus.cc:2747] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:21.155406 31877 raft_consensus.cc:1273] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:21.160544 31806 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
W20250831 07:59:21.162433 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 94617.6us
I20250831 07:59:21.143945 1757 raft_consensus.cc:2802] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:21.161047 31806 pending_rounds.cc:85] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 6
I20250831 07:59:21.164600 31726 consensus_queue.cc:260] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:21.175479 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 133933us
I20250831 07:59:21.175694 1528 raft_consensus.cc:2802] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:21.176394 1528 raft_consensus.cc:695] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:21.176863 31773 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:21.177404 1528 consensus_queue.cc:237] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.178155 31773 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:21.180182 1401 consensus_queue.cc:1046] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:59:21.181288 1401 raft_consensus.cc:2747] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:21.182056 1401 raft_consensus.cc:3053] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:21.182458 1401 raft_consensus.cc:738] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:21.143945 1668 raft_consensus.cc:695] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 7 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:21.188575 1757 raft_consensus.cc:695] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:21.188799 1668 consensus_queue.cc:237] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.189890 1757 consensus_queue.cc:237] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.157321 1731 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:21.192212 1731 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.193841 1648 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.194633 1731 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:21.196633 1648 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:21.200290 1669 raft_consensus.cc:3053] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:21.200732 1669 raft_consensus.cc:738] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:21.201591 1669 consensus_queue.cc:260] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.202224 31698 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:21.202641 1669 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:21.205099 1735 consensus_queue.cc:1035] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:21.205735 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.206676 32469 raft_consensus.cc:2391] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:21.206893 1520 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:21.207614 1520 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.171785 31806 pending_rounds.cc:107] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250831 07:59:21.213039 1520 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:21.213554 31806 raft_consensus.cc:2887] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:21.213831 1520 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:21.214366 1520 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.207871 1737 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:21.216562 1737 raft_consensus.cc:695] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:21.216749 31847 leader_election.cc:400] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5.
I20250831 07:59:21.217198 31847 leader_election.cc:403] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250831 07:59:21.168175 1498 raft_consensus.cc:2762] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250831 07:59:21.220561 31874 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:21.228083 31731 raft_consensus.cc:1273] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20250831 07:59:21.232255 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 217) took 1409 ms. Trace:
I20250831 07:59:21.232738 31874 rpcz_store.cc:276] 0831 07:59:19.813174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.914319 (+1101145us) service_pool.cc:225] Handling call
0831 07:59:21.222088 (+307769us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:21.235402 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43428 (request call id 275) took 1207 ms. Trace:
I20250831 07:59:21.235757 31803 rpcz_store.cc:276] 0831 07:59:20.027735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:20.546772 (+519037us) service_pool.cc:225] Handling call
0831 07:59:21.235219 (+688447us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b5400085cd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:21.235349 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":60979072}
I20250831 07:59:21.183535 1401 consensus_queue.cc:260] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.237421 1401 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:21.237504 31803 raft_consensus.cc:1273] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:21.239967 1306 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250831 07:59:21.242348 1585 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:21.242640 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1670) took 2732 ms. Trace:
I20250831 07:59:21.242919 1585 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:21.244725 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 65196.9us
I20250831 07:59:21.228842 31731 pending_rounds.cc:85] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 3
I20250831 07:59:21.246876 31731 pending_rounds.cc:107] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 5.4
W20250831 07:59:21.245605 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 76688.6us
I20250831 07:59:21.247275 31731 raft_consensus.cc:2887] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:21.243062 31881 rpcz_store.cc:276] 0831 07:59:18.339395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:18.755114 (+415719us) service_pool.cc:225] Handling call
0831 07:59:20.967845 (+2212731us) spinlock_profiling.cc:243] Waited 6.19 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:20.995638 (+ 27793us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:20.996066 (+ 428us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 07:59:20.996416 (+ 350us) log.cc:844] Serialized 14 byte log entry
0831 07:59:21.067548 (+ 71132us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:21.067684 (+ 136us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:21.067731 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:21.067790 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:21.071717 (+ 3927us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":3626,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":53,"raft.queue_time_us":13745,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":7270,"spinlock_wait_cycles":14233984,"thread_start_us":56993,"threads_started":1,"wal-append.queue_time_us":119397}
W20250831 07:59:21.249702 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 118131us
I20250831 07:59:21.146303 1672 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.217597 1737 consensus_queue.cc:237] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.262046 31880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1260) took 1793 ms. Trace:
I20250831 07:59:21.262991 31877 raft_consensus.cc:1238] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 07:59:21.263149 31880 rpcz_store.cc:276] 0831 07:59:19.456630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.074854 (+1618224us) service_pool.cc:225] Handling call
0831 07:59:21.161733 (+ 86879us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:21.225330 (+ 63597us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:21.225381 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:21.245223 (+ 19842us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:21.250063 (+ 4840us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:21.250126 (+ 63us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:21.250241 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:21.263945 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 61871us
I20250831 07:59:21.233175 1731 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5.
I20250831 07:59:21.207031 1735 consensus_queue.cc:1035] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.054s
I20250831 07:59:21.272620 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.273074 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.273747 31809 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:21.278878 32468 raft_consensus.cc:2391] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 8: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:21.280108 31699 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:21.290122 32465 raft_consensus.cc:1273] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:21.296128 1606 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:21.298091 1523 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:21.299098 1303 raft_consensus.cc:2747] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:21.298884 1523 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.299311 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.300362 31699 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:21.301206 1303 raft_consensus.cc:2747] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:21.301318 1523 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:21.301679 31699 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:21.298498 31733 raft_consensus.cc:2408] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:21.302542 1498 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:21.303421 1498 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.304035 1740 raft_consensus.cc:2747] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:21.305087 31699 leader_election.cc:400] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6.
I20250831 07:59:21.306334 1498 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:20.880599 31727 rpcz_store.cc:276] 0831 07:59:18.639690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:18.643966 (+ 4276us) service_pool.cc:225] Handling call
0831 07:59:18.644358 (+ 392us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 07:59:19.011282 (+366924us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:19.011313 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 07:59:19.033151 (+ 21838us) log.cc:844] Serialized 52 byte log entry
0831 07:59:19.109833 (+ 76682us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:19.122687 (+ 12854us) log.cc:844] Serialized 14 byte log entry
0831 07:59:19.929671 (+806984us) spinlock_profiling.cc:243] Waited 5.96 ms on lock 0x7b180037fcc0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce5069786 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcecebfc41 00007fdceceb6de7 00007fdceceb6c7a 00007fdcecfac06f
0831 07:59:20.781052 (+851381us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:20.781231 (+ 179us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:20.781289 (+ 58us) raft_consensus.cc:1683] finished
0831 07:59:20.798409 (+ 17120us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:20.802353 (+ 3944us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1502317,"prepare.queue_time_us":10677,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":1606589,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":128,"spinlock_wait_cycles":88755840,"thread_start_us":243457,"threads_started":3,"wal-append.queue_time_us":230736,"wal-append.run_cpu_time_us":4514,"wal-append.run_wall_time_us":1580665}
I20250831 07:59:21.307786 1585 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:21.312726 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.314633 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:21.314805 32470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47886 (request call id 405) took 3691 ms (client timeout 1532 ms). Trace:
I20250831 07:59:21.315354 31727 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
W20250831 07:59:21.315215 32470 rpcz_store.cc:269] 0831 07:59:17.623328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.623622 (+ 294us) service_pool.cc:225] Handling call
0831 07:59:19.674756 (+2051134us) spinlock_profiling.cc:243] Waited 5.89 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:21.314599 (+1639843us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540062f054. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:21.314722 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2787474432}
I20250831 07:59:21.316176 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:21.317584 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 53727.3us
I20250831 07:59:21.324272 1018 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:21.325631 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:21.281280 32435 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
W20250831 07:59:21.326795 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:21.327296 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:21.328576 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 120573us
I20250831 07:59:21.328971 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:21.327193 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 46028.1us
I20250831 07:59:21.330564 31731 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:21.275776 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.278237 1662 raft_consensus.cc:2762] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20250831 07:59:21.275605 1683 consensus_queue.cc:1162] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8247a38b550649aa9cd6e7668dcaec10" responder_term: 5 status { last_received { term: 3 index: 2 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250831 07:59:21.348304 1303 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:21.348717 1625 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:21.349406 1625 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.349335 1303 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.350414 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1275) took 1782 ms. Trace:
I20250831 07:59:21.350770 31733 rpcz_store.cc:276] 0831 07:59:19.567831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:19.585527 (+ 17696us) service_pool.cc:225] Handling call
0831 07:59:21.350159 (+1764632us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b540038c7d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:21.350332 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3935532928}
I20250831 07:59:21.353082 32466 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:21.277864 1735 consensus_queue.cc:1230] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 07:59:21.281926 1520 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:21.351588 1625 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:21.336418 1264 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:21.365329 31847 leader_election.cc:400] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6.
I20250831 07:59:21.365638 31847 leader_election.cc:403] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20250831 07:59:21.362954 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 99459.5us
W20250831 07:59:21.370457 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:21.375520 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1270) took 1275 ms. Trace:
W20250831 07:59:21.362421 31698 connection.cc:232] RPC call timeout handler was delayed by -1.4524012709999852s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.385s for a response
W20250831 07:59:21.378257 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:21.381297 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.381546 31809 raft_consensus.cc:1238] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250831 07:59:21.380590 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:21.363071 1498 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:21.397456 1740 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:21.397099 1520 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250831 07:59:21.397646 1498 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.398046 1740 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.382437 1625 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:21.400287 1625 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.402225 1625 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:21.403367 31698 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:21.404047 31804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1567) took 3928 ms (client timeout 1768 ms). Trace:
W20250831 07:59:21.404454 31804 rpcz_store.cc:269] 0831 07:59:17.475466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.484821 (+ 9355us) service_pool.cc:225] Handling call
0831 07:59:21.362998 (+3878177us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b5400220454. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:21.403867 (+ 40869us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":416721280}
I20250831 07:59:21.404913 1275 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:21.405766 1275 consensus_queue.cc:1046] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:59:21.406734 1275 raft_consensus.cc:3053] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:21.407136 1275 raft_consensus.cc:738] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:21.408515 1672 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:21.408217 1275 consensus_queue.cc:260] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.409776 1275 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:21.387961 31699 connection.cc:232] RPC call timeout handler was delayed by -2.6528906620000043s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1768s for a response
I20250831 07:59:21.412019 1735 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:21.412532 1735 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.413085 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.414381 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.414635 1735 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:21.415499 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:21.416178 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:21.416592 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.416622 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:21.417430 31732 raft_consensus.cc:2391] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:21.417564 31727 raft_consensus.cc:2391] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 6: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:21.417675 32464 raft_consensus.cc:1238] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:21.418215 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:21.418349 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:21.418414 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.418756 1606 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250831 07:59:21.419206 31882 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:21.420085 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.420239 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:21.420847 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.421264 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.421468 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.422889 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.423019 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.423511 31734 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:21.390739 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1562) took 3916 ms. Trace:
I20250831 07:59:21.391031 1624 consensus_queue.cc:1035] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20250831 07:59:21.394526 1303 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:21.395395 31848 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:21.383987 1264 raft_consensus.cc:695] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:21.397253 31847 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:21.428597 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 52606.1us
I20250831 07:59:21.430059 31847 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:21.376274 31883 rpcz_store.cc:276] 0831 07:59:20.089775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.089870 (+1000095us) service_pool.cc:225] Handling call
0831 07:59:21.099998 (+ 10128us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:21.136708 (+ 36710us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 07:59:21.162733 (+ 26025us) log.cc:844] Serialized 14 byte log entry
0831 07:59:21.363735 (+201002us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:21.363833 (+ 98us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:21.364351 (+ 518us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:21.364464 (+ 113us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:21.364628 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38759,"prepare.queue_time_us":18060,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"raft.queue_time_us":38456,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":28763,"thread_start_us":151118,"threads_started":1}
I20250831 07:59:21.416790 31731 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:21.433665 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.434401 32470 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:21.434785 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.435600 1306 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:21.436182 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.436151 1306 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.399705 1498 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:21.424187 31805 rpcz_store.cc:276] 0831 07:59:17.473956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:17.481725 (+ 7769us) service_pool.cc:225] Handling call
0831 07:59:21.388981 (+3907256us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:21.389426 (+ 445us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:21.389453 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:21.389507 (+ 54us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:21.389533 (+ 26us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:21.389604 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:21.390415 (+ 811us) spinlock_profiling.cc:243] Waited 3.87 s on lock 0x7b5400220450. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:21.390555 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":772,"spinlock_wait_cycles":8974749824}
I20250831 07:59:21.438297 1306 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:21.440042 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.440268 1686 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:21.440732 32464 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:59:21.441766 1396 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250831 07:59:21.442795 31808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39068 (request call id 614) took 1733 ms. Trace:
I20250831 07:59:21.443423 31725 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:21.380596 1666 raft_consensus.cc:2747] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6.
I20250831 07:59:21.444903 1264 consensus_queue.cc:237] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.445859 31726 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:21.446445 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.448348 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.448897 31734 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:21.440868 1686 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:21.443187 31808 rpcz_store.cc:276] 0831 07:59:19.709178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.288573 (+1579395us) service_pool.cc:225] Handling call
0831 07:59:21.442049 (+153476us) spinlock_profiling.cc:243] Waited 95.6 ms on lock 0x7b54002f0654. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f81cd0 00007fdce8005eea 00007fdcecf5e60e 00007fdce65fbcc7 00007fdce65fbc52 00007fdce65fbb81 00007fdce65fbafc 00007fdce65fadb2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:21.442718 (+ 669us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":219839744}
I20250831 07:59:21.451213 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:21.464546 1275 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:21.464912 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.465384 1275 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.466709 31733 raft_consensus.cc:1273] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250831 07:59:21.467084 31733 pending_rounds.cc:85] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 2
I20250831 07:59:21.467303 31733 pending_rounds.cc:107] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250831 07:59:21.467542 31733 raft_consensus.cc:2887] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:21.467561 1275 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:21.469187 1508 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.472048 32469 raft_consensus.cc:1238] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:59:21.472952 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:21.471268 1508 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:21.489346 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.474066 1686 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:21.475486 1498 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:21.484098 1600 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:21.488124 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.488170 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.491564 31698 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:21.497594 31698 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:21.489979 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.494417 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:21.495631 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:21.502324 31881 raft_consensus.cc:2408] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20250831 07:59:21.504163 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 108857us
I20250831 07:59:21.505833 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:21.480767 1740 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:21.508476 31880 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:21.512117 32470 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:21.479562 1303 consensus_queue.cc:1046] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20250831 07:59:21.503623 1625 raft_consensus.cc:2802] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:21.503789 1275 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:21.533383 31848 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:21.533600 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.535326 31731 raft_consensus.cc:2408] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250831 07:59:21.529430 1625 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:21.541628 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.530189 1275 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:21.551016 31882 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:21.552836 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.554199 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:21.506968 1498 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.555913 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1265) took 1468 ms. Trace:
I20250831 07:59:21.532342 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:21.538434 1303 consensus_queue.cc:1230] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 7ecb257174554a26b2c35ad04b0eabae log is divergent from this leader: its last log entry 5.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250831 07:59:21.542763 1625 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:21.561393 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1273) took 1470 ms. Trace:
I20250831 07:59:21.549167 1438 raft_consensus.cc:2802] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:21.563417 32434 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:21.557308 31882 rpcz_store.cc:276] 0831 07:59:20.083977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.027618 (+943641us) service_pool.cc:225] Handling call
0831 07:59:21.418213 (+390595us) spinlock_profiling.cc:243] Waited 4.41 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:21.551981 (+133768us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10137344}
I20250831 07:59:21.552750 1275 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:21.570847 31773 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:21.566076 31875 rpcz_store.cc:276] 0831 07:59:20.090757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.101715 (+1010958us) service_pool.cc:225] Handling call
0831 07:59:21.102131 (+ 416us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:21.102461 (+ 330us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 07:59:21.103324 (+ 863us) log.cc:844] Serialized 14 byte log entry
0831 07:59:21.560822 (+457498us) log.cc:844] Serialized 14 byte log entry
0831 07:59:21.560964 (+ 142us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:21.561007 (+ 43us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:21.561049 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:21.561231 (+ 182us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:21.561341 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8429,"prepare.queue_time_us":5219,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"thread_start_us":448016,"threads_started":1}
W20250831 07:59:21.577226 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:21.575170 31773 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:21.549389 1666 consensus_queue.cc:1035] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.022s
I20250831 07:59:21.584371 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:21.580947 32436 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.974s (SENT) [suppressed 41 similar messages]
I20250831 07:59:21.585125 31880 raft_consensus.cc:2391] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:21.579921 1757 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:21.585937 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.586222 1757 raft_consensus.cc:695] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:21.586665 31875 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
I20250831 07:59:21.571559 1438 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:21.573284 1378 raft_consensus.cc:2802] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:21.589560 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:21.580564 1669 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:21.579978 32466 raft_consensus.cc:1238] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:21.564560 1538 consensus_queue.cc:1046] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20250831 07:59:21.599377 31773 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:21.602068 31880 raft_consensus.cc:1238] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:21.604537 1498 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:21.605846 31875 raft_consensus.cc:1238] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 8. Current term is 9. Ops: [4.3-8.5]
I20250831 07:59:21.606885 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.607683 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:21.610738 1771 log.cc:927] Time spent T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250831 07:59:21.616236 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:21.616380 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.587219 1401 raft_consensus.cc:2802] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:21.635233 31883 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:21.599457 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.615418 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.637305 31698 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:21.638608 31847 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:21.594167 1378 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:21.595250 1669 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:21.628638 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.646718 31698 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:21.647053 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 47779us
I20250831 07:59:21.592690 1438 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:21.637250 31728 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:21.649748 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:21.628820 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.628638 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.615913 1757 consensus_queue.cc:237] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:21.643741 31849 leader_election.cc:336] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.595s (SENT)
I20250831 07:59:21.649856 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.642469 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:21.650813 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:21.648051 31773 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:21.605873 32462 raft_consensus.cc:1273] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250831 07:59:21.688809 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 89716.7us
I20250831 07:59:21.690920 31802 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:21.692766 31809 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:21.691568 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:21.659765 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.694341 31808 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:21.696563 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.650983 31878 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:21.698240 31729 raft_consensus.cc:2408] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250831 07:59:21.694653 31849 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:21.700497 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 53873.1us
I20250831 07:59:21.699793 31734 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:21.691938 31732 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:21.646158 1378 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:21.638043 1401 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:21.709872 31698 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:21.648679 1669 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:21.750264 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 59073.9us
I20250831 07:59:21.750386 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:21.657610 1498 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:21.644498 1662 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:21.756321 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 108414us
I20250831 07:59:21.757968 31807 raft_consensus.cc:2391] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:21.706180 1740 raft_consensus.cc:2747] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:21.760562 32434 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:21.773989 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.774775 31729 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:21.707952 1668 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:21.778086 31698 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:21.736145 1401 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:21.755995 1577 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:21.757047 1585 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:21.786550 31773 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:21.789757 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 151139us
I20250831 07:59:21.798698 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.767571 1737 raft_consensus.cc:2802] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:21.780561 1668 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:21.782639 1240 raft_consensus.cc:2747] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:21.806139 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:21.807267 31726 raft_consensus.cc:2391] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 6: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
W20250831 07:59:21.809456 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 165741us
I20250831 07:59:21.819020 31807 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
W20250831 07:59:21.821235 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:21.790758 1264 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:21.826464 1785 consensus_queue.cc:1046] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:59:21.833001 31809 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 9.
I20250831 07:59:21.833683 1596 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20250831 07:59:21.834214 1773 log.cc:927] Time spent T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20250831 07:59:21.835903 1776 log.cc:927] Time spent T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20250831 07:59:21.837159 1770 log.cc:927] Time spent T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.235s user 0.001s sys 0.000s
I20250831 07:59:21.805748 1737 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:21.837999 1737 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:21.842725 31773 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:21.849920 1662 raft_consensus.cc:2747] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250831 07:59:21.811319 1525 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20250831 07:59:21.804647 1498 raft_consensus.cc:2802] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:21.804826 1668 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:21.827642 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:21.782859 1303 consensus_queue.cc:1046] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.036s
I20250831 07:59:21.861580 1577 raft_consensus.cc:2802] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:21.873383 31773 leader_election.cc:400] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585) with higher term. Message: Invalid argument: T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7.
I20250831 07:59:21.873823 31773 leader_election.cc:403] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250831 07:59:21.881747 1683 consensus_queue.cc:1046] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250831 07:59:21.882694 1683 consensus_queue.cc:1230] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 7ecb257174554a26b2c35ad04b0eabae log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250831 07:59:21.883324 1585 raft_consensus.cc:2747] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585) with higher term. Message: Invalid argument: T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7.
I20250831 07:59:21.890899 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:21.862915 1498 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:21.898379 31729 raft_consensus.cc:1273] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:21.866147 1264 raft_consensus.cc:695] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:21.904456 31805 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250831 07:59:21.908496 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.909256 31726 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
W20250831 07:59:21.910075 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 47520.5us
I20250831 07:59:21.912201 1585 consensus_queue.cc:1046] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:21.927312 1737 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.929765 1737 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:21.930083 1401 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:21.932083 1018 consensus_queue.cc:1035] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:59:21.933478 1018 consensus_queue.cc:1035] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:21.876192 1624 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.064s
I20250831 07:59:21.927914 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:21.904578 1498 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:21.935592 1775 log.cc:927] Time spent T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.124s user 0.001s sys 0.001s
W20250831 07:59:21.965379 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:21.941773 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.942166 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.940423 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:21.940800 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:21.940019 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:21.938266 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.936159 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:22.021950 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:22.025885 31804 raft_consensus.cc:1811] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Handling vote request from an unknown peer a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:21.937667 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:21.905267 1577 raft_consensus.cc:695] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:21.895469 1303 consensus_queue.cc:1046] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.122s
I20250831 07:59:21.941377 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:22.025101 31733 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:22.032789 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:22.036064 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:22.036936 31804 raft_consensus.cc:2391] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:22.034274 31806 raft_consensus.cc:1238] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250831 07:59:22.044962 1303 consensus_queue.cc:1230] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 07:59:22.046427 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:22.047201 31730 raft_consensus.cc:2391] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:22.047252 31800 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:22.045742 31802 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:22.034556 31734 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:21.937592 1735 raft_consensus.cc:3053] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:21.942786 1600 raft_consensus.cc:3053] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 07:59:21.935490 1275 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:22.067035 32436 leader_election.cc:336] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.518s (SENT)
I20250831 07:59:22.056691 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:22.079572 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:22.057802 31802 raft_consensus.cc:1238] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 07:59:22.081406 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:22.075173 32436 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:22.103912 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:22.105882 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:22.020936 1306 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20250831 07:59:22.105876 31802 raft_consensus.cc:1238] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250831 07:59:21.920070 1662 consensus_queue.cc:1035] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.159s
I20250831 07:59:22.056403 1735 raft_consensus.cc:738] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:21.936431 940 consensus_queue.cc:1035] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.120s
I20250831 07:59:22.111040 31731 raft_consensus.cc:1238] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 6. Current term is 7. Ops: []
W20250831 07:59:22.080649 1686 consensus_meta.cc:287] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.552s user 0.005s sys 0.005s
I20250831 07:59:21.931728 1625 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:22.113668 31806 raft_consensus.cc:1238] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:22.115942 31802 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 07:59:22.056274 1600 raft_consensus.cc:738] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:22.059474 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:22.033788 1438 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:22.101475 1508 raft_consensus.cc:2747] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:22.059939 1577 consensus_queue.cc:237] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:21.940326 1264 consensus_queue.cc:237] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:22.100196 1495 raft_consensus.cc:3053] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 8 LEADER]: Stepping down as leader of term 8
W20250831 07:59:22.396709 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 70301.5us
I20250831 07:59:22.289203 31733 raft_consensus.cc:1273] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 07:59:22.468075 31875 raft_consensus.cc:1238] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
W20250831 07:59:22.451443 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:22.330861 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 404199us
I20250831 07:59:22.211505 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:22.450320 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 383332us
I20250831 07:59:22.450974 31732 raft_consensus.cc:1273] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20250831 07:59:22.396225 31698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.008s sys 0.016s
W20250831 07:59:22.474462 31807 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.019s sys 0.049s
W20250831 07:59:22.475756 31807 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s user 0.019s sys 0.049s
I20250831 07:59:22.450197 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:22.284235 31801 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.000s sys 0.025s
W20250831 07:59:22.485445 31801 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s user 0.000s sys 0.033s
I20250831 07:59:22.477425 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1744) took 1044 ms. Trace:
I20250831 07:59:22.218775 31808 raft_consensus.cc:1238] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 8. Current term is 9. Ops: [4.3-8.5]
I20250831 07:59:22.486465 31725 rpcz_store.cc:276] 0831 07:59:21.415105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.415256 (+ 151us) service_pool.cc:225] Handling call
0831 07:59:22.450426 (+1035170us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b540038d1d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:22.459140 (+ 8714us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":45724544}
I20250831 07:59:22.205350 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:22.212164 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:22.474753 31698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.603s user 0.009s sys 0.021s
W20250831 07:59:22.488758 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 617718us
I20250831 07:59:22.490391 31698 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:22.487795 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:22.317745 1669 consensus_meta.cc:287] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.549s user 0.005s sys 0.004s
W20250831 07:59:22.323015 1777 log.cc:927] Time spent T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.209s user 0.000s sys 0.000s
I20250831 07:59:22.513003 31729 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 6: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:22.524941 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:22.537091 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 57710.3us
I20250831 07:59:22.537977 31775 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:22.198064 1668 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:22.539149 31775 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:22.540129 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:22.540335 1668 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:22.540670 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 50370.1us
I20250831 07:59:22.540832 31734 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:22.541620 31698 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:22.542665 31698 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:22.544170 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:22.544847 31725 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:22.201105 1401 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:22.537163 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:22.290509 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:22.325664 1625 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:22.551851 1585 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.015s sys 0.045s
I20250831 07:59:22.557421 31877 raft_consensus.cc:2391] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:22.563145 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:22.565004 1438 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:22.474802 1495 raft_consensus.cc:738] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:22.568118 1600 consensus_queue.cc:260] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:22.569123 1672 raft_consensus.cc:2747] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:22.587808 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:22.578155 31848 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:22.586133 31849 leader_election.cc:336] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.665s (SENT)
W20250831 07:59:22.511459 1378 consensus_meta.cc:287] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.742s user 0.011s sys 0.000s
I20250831 07:59:22.596637 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:22.198263 1603 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.395s
I20250831 07:59:22.595556 31730 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
I20250831 07:59:22.199563 1735 consensus_queue.cc:260] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:22.606770 31848 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:22.572031 1662 raft_consensus.cc:2747] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:22.608665 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:22.587235 1585 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.664s user 0.016s sys 0.045s
I20250831 07:59:22.614733 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:22.556753 1275 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:22.624974 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:22.571286 1740 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:22.616966 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:22.572957 1737 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:22.626852 31848 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:22.630220 31848 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:22.621384 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 73145.9us
I20250831 07:59:22.622965 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:22.632922 31875 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:22.518129 1669 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:22.160315 1686 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:22.626776 31803 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.005s sys 0.019s
W20250831 07:59:22.608525 1596 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s user 0.026s sys 0.061s
I20250831 07:59:22.620108 31775 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:22.608651 1625 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:22.605916 1378 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:22.644909 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:22.645197 31775 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:22.642751 1596 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s user 0.026s sys 0.062s
I20250831 07:59:22.647631 32434 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:22.625366 1600 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:59:22.654091 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:22.648687 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:22.652920 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 66885.4us
I20250831 07:59:22.629551 1735 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:22.651984 1731 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:22.658386 1731 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:22.635337 1740 raft_consensus.cc:695] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:22.655617 1367 raft_consensus.cc:2802] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:22.655974 1306 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 07:59:22.656530 31874 raft_consensus.cc:2366] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:22.653209 32434 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:22.660823 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:22.647019 1625 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:22.647104 1668 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 07:59:22.667147 31730 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:22.675067 32435 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.511s user 0.015s sys 0.015s
I20250831 07:59:22.649974 1438 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:22.642975 31803 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.608s user 0.006s sys 0.019s
W20250831 07:59:22.694500 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 531038us
I20250831 07:59:22.652205 1528 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 07:59:22.671640 31726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.008s sys 0.012s
I20250831 07:59:22.665179 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:22.692260 31875 raft_consensus.cc:2391] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
W20250831 07:59:22.671283 31727 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.005s sys 0.014s
W20250831 07:59:22.698621 31727 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.552s user 0.005s sys 0.014s
W20250831 07:59:22.699051 31726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s user 0.008s sys 0.012s
W20250831 07:59:22.716017 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 62884.5us
I20250831 07:59:22.716862 31734 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 6: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
W20250831 07:59:22.721344 31849 leader_election.cc:336] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.654s (SENT)
W20250831 07:59:22.885000 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 163683us
W20250831 07:59:22.885982 31849 leader_election.cc:336] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.864s (SENT)
I20250831 07:59:22.657083 1737 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:22.894363 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 249229us
I20250831 07:59:22.667152 1367 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:22.692171 1797 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:22.694697 1625 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:22.901271 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:22.919628 31775 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:22.901786 1498 consensus_meta.cc:287] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.853s user 0.010s sys 0.001s
I20250831 07:59:22.930541 31734 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:22.890515 1668 raft_consensus.cc:695] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 9 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:22.894848 1798 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:22.893005 1538 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:22.891351 1596 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:22.967130 1596 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:22.944764 1796 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:22.967700 1596 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:22.968075 1796 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:22.968541 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:22.898074 1585 raft_consensus.cc:2747] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:22.902595 1367 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:22.939441 31733 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:22.736054 1438 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:22.945544 1498 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:22.955291 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:22.913519 1797 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:22.989799 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:22.980953 31809 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.023s sys 0.038s
W20250831 07:59:22.990772 31809 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.510s user 0.023s sys 0.039s
W20250831 07:59:22.986546 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 440453us
I20250831 07:59:22.690503 1495 consensus_queue.cc:260] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:22.903798 1737 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:22.956780 1789 log.cc:927] Time spent T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20250831 07:59:23.063657 1784 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:22.955438 1787 raft_consensus.cc:2802] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 07:59:23.087395 31879 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.017s sys 0.034s
I20250831 07:59:23.093168 31728 raft_consensus.cc:2366] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5.
W20250831 07:59:23.088743 32436 leader_election.cc:336] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.887s (SENT)
W20250831 07:59:23.094579 31879 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s user 0.017s sys 0.035s
I20250831 07:59:22.961910 1731 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:23.099139 31883 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.023s sys 0.030s
I20250831 07:59:22.993412 1797 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:23.120170 31883 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s user 0.023s sys 0.033s
I20250831 07:59:22.988338 1538 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:23.176605 31697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.022s sys 0.036s
W20250831 07:59:23.254230 31697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.667s user 0.027s sys 0.037s
W20250831 07:59:23.118173 31700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.017s sys 0.035s
W20250831 07:59:23.255291 31700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.682s user 0.017s sys 0.048s
W20250831 07:59:23.255960 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:23.233201 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 223713us
I20250831 07:59:22.992964 1740 consensus_queue.cc:237] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:23.114059 1788 log.cc:927] Time spent T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20250831 07:59:23.094441 1585 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:22.989168 1798 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.116472 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:23.210480 1789 log.cc:927] Time spent T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
W20250831 07:59:23.232434 1794 log.cc:927] Time spent T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.287s user 0.000s sys 0.000s
I20250831 07:59:23.268098 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:23.091120 1686 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:23.269065 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 383107us
W20250831 07:59:23.268561 1577 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.697s user 0.002s sys 0.008s
I20250831 07:59:23.270841 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:23.099768 1787 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:59:23.279117 31730 consensus_meta.cc:287] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.551s user 0.007s sys 0.000s
W20250831 07:59:23.282344 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 378083us
I20250831 07:59:23.290102 31730 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:23.290691 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 205) took 1081 ms. Trace:
I20250831 07:59:23.233342 1498 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:23.232704 1804 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:23.252959 1396 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:23.115097 1792 log.cc:927] Time spent T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.178s user 0.000s sys 0.000s
W20250831 07:59:23.272142 1547 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.008s sys 0.024s
I20250831 07:59:23.270685 1585 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:23.291056 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:23.290968 31734 raft_consensus.cc:1273] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250831 07:59:23.287024 31849 leader_election.cc:336] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.002s (SENT)
I20250831 07:59:23.290987 31730 rpcz_store.cc:276] 0831 07:59:22.209240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:22.649390 (+440150us) service_pool.cc:225] Handling call
0831 07:59:23.290634 (+641244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:23.296231 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:23.296545 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:23.128393 1784 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.318390 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:23.319531 1784 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:23.291427 1787 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:23.299439 1538 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:23.302837 1275 consensus_queue.cc:1046] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:23.304261 32462 raft_consensus.cc:1238] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:59:23.231948 1668 consensus_queue.cc:237] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.306229 31805 raft_consensus.cc:1215] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 3.6->[4.7-4.7] Dedup: 4.7->[]
W20250831 07:59:23.306437 31876 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.011s sys 0.040s
W20250831 07:59:23.332432 31876 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s user 0.011s sys 0.040s
I20250831 07:59:23.207165 1669 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:23.289585 1577 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s user 0.002s sys 0.009s
I20250831 07:59:23.314465 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 292) took 1452 ms. Trace:
I20250831 07:59:23.288595 32469 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 5
W20250831 07:59:23.305610 1795 log.cc:927] Time spent T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.218s user 0.000s sys 0.001s
I20250831 07:59:23.339200 31805 raft_consensus.cc:1238] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20250831 07:59:23.339919 31809 rpcz_store.cc:276] 0831 07:59:21.853520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.853679 (+ 159us) service_pool.cc:225] Handling call
0831 07:59:21.854143 (+ 464us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 07:59:21.862043 (+ 7900us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:21.862089 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 07:59:21.868512 (+ 6423us) log.cc:844] Serialized 52 byte log entry
0831 07:59:22.446640 (+578128us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:22.991248 (+544608us) log.cc:844] Serialized 14 byte log entry
0831 07:59:22.991486 (+ 238us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:22.991567 (+ 81us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:23.305786 (+314219us) raft_consensus.cc:1683] finished
0831 07:59:23.305809 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:23.305952 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1213,"prepare.queue_time_us":652190,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":219441,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":13221,"thread_start_us":764731,"threads_started":2,"wal-append.queue_time_us":862705}
W20250831 07:59:23.310492 1547 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.021s user 0.008s sys 0.024s
I20250831 07:59:23.345487 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1299) took 1227 ms. Trace:
W20250831 07:59:23.347951 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 60963.6us
W20250831 07:59:23.349305 31849 leader_election.cc:336] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.058s (SENT)
I20250831 07:59:23.349838 31849 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:23.346050 31876 rpcz_store.cc:276] 0831 07:59:22.118210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:22.140303 (+ 22093us) service_pool.cc:225] Handling call
0831 07:59:22.627862 (+487559us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:22.628087 (+ 225us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:22.635345 (+ 7258us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 07:59:22.650553 (+ 15208us) log.cc:844] Serialized 14 byte log entry
0831 07:59:23.345001 (+694448us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:23.345078 (+ 77us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:23.345154 (+ 76us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.345243 (+ 89us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:23.345397 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":17044,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"spinlock_wait_cycles":8387328,"thread_start_us":682024,"threads_started":1}
I20250831 07:59:23.356254 31775 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:23.324309 1275 consensus_queue.cc:1230] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250831 07:59:23.313095 1585 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:23.325537 1396 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:23.367373 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1302) took 1197 ms. Trace:
I20250831 07:59:23.368012 1538 raft_consensus.cc:2747] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250831 07:59:23.368970 31877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1684) took 1828 ms (client timeout 1926 ms). Trace:
W20250831 07:59:23.382612 31877 rpcz_store.cc:269] 0831 07:59:21.541030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.541215 (+ 185us) service_pool.cc:225] Handling call
0831 07:59:23.368796 (+1827581us) spinlock_profiling.cc:243] Waited 999 ms on lock 0x7b540007dad4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:23.368918 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2298491648}
I20250831 07:59:23.383044 31809 raft_consensus.cc:1273] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 07:59:23.371767 31875 rpcz_store.cc:276] 0831 07:59:22.170110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:22.640175 (+470065us) service_pool.cc:225] Handling call
0831 07:59:23.367195 (+727020us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b54003efb54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:23.367317 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29279616}
I20250831 07:59:23.368525 1784 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:23.368160 31628 catalog_manager.cc:5582] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 2, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 2 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:23.389246 1784 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:23.389813 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 217) took 1141 ms. Trace:
I20250831 07:59:23.390199 31734 rpcz_store.cc:276] 0831 07:59:22.248253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:22.941310 (+693057us) service_pool.cc:225] Handling call
0831 07:59:23.290649 (+349339us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:23.315788 (+ 25139us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.389606 (+ 73818us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b540018fe50. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:23.389757 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":762176256}
I20250831 07:59:23.391384 1784 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:23.315058 1804 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.392021 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1583) took 1470 ms. Trace:
I20250831 07:59:23.392392 31805 rpcz_store.cc:276] 0831 07:59:21.922064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.931591 (+ 9527us) service_pool.cc:225] Handling call
0831 07:59:23.306121 (+1374530us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:23.343181 (+ 37060us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.391846 (+ 48665us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400084dd0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:23.391961 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3160263424}
W20250831 07:59:23.394620 1735 consensus_meta.cc:287] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.737s user 0.006s sys 0.000s
I20250831 07:59:23.394848 1018 raft_consensus.cc:3053] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:23.382530 1731 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:23.395347 1018 raft_consensus.cc:738] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:23.395534 1800 log.cc:927] Time spent T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20250831 07:59:23.395694 1731 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:23.386161 1438 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:23.396276 1018 consensus_queue.cc:260] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.397301 1018 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:23.397835 1731 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:23.400307 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1753) took 1861 ms. Trace:
I20250831 07:59:23.400719 31731 rpcz_store.cc:276] 0831 07:59:21.538947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.542639 (+ 3692us) service_pool.cc:225] Handling call
0831 07:59:21.550700 (+ 8061us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:22.118098 (+567398us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.400053 (+1281955us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b5400021a54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:23.400231 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1288242560}
I20250831 07:59:23.402503 1683 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:23.394424 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:23.342964 1378 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:23.402980 1683 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:23.409410 31803 raft_consensus.cc:1273] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 07:59:23.409544 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 289) took 1765 ms. Trace:
I20250831 07:59:23.353039 1737 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:23.436627 31875 raft_consensus.cc:1238] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: []
W20250831 07:59:23.436590 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 52534.2us
I20250831 07:59:23.437268 31876 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250831 07:59:23.442200 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:23.442772 31877 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:23.439325 31801 rpcz_store.cc:276] 0831 07:59:21.644019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.648938 (+ 4919us) service_pool.cc:225] Handling call
0831 07:59:21.701740 (+ 52802us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:21.705014 (+ 3274us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:21.705059 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:21.705680 (+ 621us) log.cc:844] Serialized 26 byte log entry
0831 07:59:22.485947 (+780267us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:22.485986 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:22.486050 (+ 64us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:23.409334 (+923284us) raft_consensus.cc:1683] finished
0831 07:59:23.409356 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:23.409489 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2813,"raft.queue_time_us":3758,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":24457,"thread_start_us":711876,"threads_started":1,"wal-append.queue_time_us":1147145}
I20250831 07:59:23.441684 31809 raft_consensus.cc:1238] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 1. Current term is 2. Ops: []
I20250831 07:59:23.362221 1796 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:23.449349 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:23.387113 1662 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:23.404419 1525 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:23.474372 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 131312us
I20250831 07:59:23.475982 31775 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:23.474570 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1777) took 1535 ms. Trace:
I20250831 07:59:23.478878 31806 raft_consensus.cc:1273] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:23.480626 31882 raft_consensus.cc:1238] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 7 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20250831 07:59:23.481159 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 223205us
I20250831 07:59:23.481429 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1688) took 1879 ms. Trace:
I20250831 07:59:23.481930 31882 rpcz_store.cc:276] 0831 07:59:21.601837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.615906 (+ 14069us) service_pool.cc:225] Handling call
0831 07:59:23.480130 (+1864224us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:23.480324 (+ 194us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:23.481130 (+ 806us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.481363 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2528640}
I20250831 07:59:23.478502 31726 rpcz_store.cc:276] 0831 07:59:21.939613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:22.042426 (+102813us) service_pool.cc:225] Handling call
0831 07:59:22.045633 (+ 3207us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:22.046075 (+ 442us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:22.046105 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:22.046574 (+ 469us) log.cc:844] Serialized 26 byte log entry
0831 07:59:22.779716 (+733142us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:23.291514 (+511798us) log.cc:844] Serialized 14 byte log entry
0831 07:59:23.474128 (+182614us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.474243 (+ 115us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:23.474303 (+ 60us) raft_consensus.cc:1683] finished
0831 07:59:23.474324 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:23.474491 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":422646,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":116581,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":22435,"thread_start_us":1146517,"threads_started":3,"wal-append.queue_time_us":1121781}
I20250831 07:59:23.483417 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1780) took 1373 ms. Trace:
I20250831 07:59:23.483913 31732 rpcz_store.cc:276] 0831 07:59:22.113963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:22.121564 (+ 7601us) service_pool.cc:225] Handling call
0831 07:59:22.450555 (+328991us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:22.473540 (+ 22985us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.483175 (+1009635us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b540038d1d0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:23.483344 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":574168448}
I20250831 07:59:23.352887 1596 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:23.489409 32436 leader_election.cc:336] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.010s (SENT)
I20250831 07:59:23.444031 1275 raft_consensus.cc:3053] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:23.496182 32436 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:23.488221 31775 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:23.508899 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:23.517083 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 218) took 1070 ms. Trace:
I20250831 07:59:23.520430 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:23.521042 31726 raft_consensus.cc:2374] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4. Re-sending same reply.
I20250831 07:59:23.469161 1662 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:23.517467 31733 rpcz_store.cc:276] 0831 07:59:22.447114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:22.993767 (+546653us) service_pool.cc:225] Handling call
0831 07:59:23.253168 (+259401us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:23.268983 (+ 15815us) raft_consensus.cc:1532] Early marking committed up to index 6
0831 07:59:23.287728 (+ 18745us) log.cc:844] Serialized 14 byte log entry
0831 07:59:23.516775 (+229047us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:23.516826 (+ 51us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:23.516877 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.516938 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:23.517032 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56840,"prepare.queue_time_us":18542,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"thread_start_us":171663,"threads_started":1}
W20250831 07:59:23.520677 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 161963us
W20250831 07:59:23.535465 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 46077.9us
I20250831 07:59:23.490669 1796 raft_consensus.cc:2802] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250831 07:59:23.532707 31727 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.010s sys 0.032s
W20250831 07:59:23.540017 32436 leader_election.cc:336] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.110s (SENT)
W20250831 07:59:23.540215 31727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s user 0.010s sys 0.032s
W20250831 07:59:23.541318 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 53180.8us
I20250831 07:59:23.542181 31775 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:23.543243 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:23.543298 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:23.544031 31803 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:23.544107 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1776) took 1604 ms. Trace:
I20250831 07:59:23.538304 31773 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:23.453739 1757 consensus_queue.cc:1046] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
I20250831 07:59:23.510602 1275 raft_consensus.cc:738] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:23.558575 31734 raft_consensus.cc:3053] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:23.563705 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 225) took 1283 ms. Trace:
I20250831 07:59:23.454233 1711 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20250831 07:59:23.548602 31727 rpcz_store.cc:276] 0831 07:59:21.939321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:22.040188 (+100867us) service_pool.cc:225] Handling call
0831 07:59:22.078568 (+ 38380us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:22.112707 (+ 34139us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 07:59:22.719455 (+606748us) log.cc:844] Serialized 14 byte log entry
0831 07:59:23.542478 (+823023us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:23.542530 (+ 52us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:23.543724 (+ 1194us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.543852 (+ 128us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:23.544017 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":976983,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":96,"thread_start_us":1157996,"threads_started":2}
I20250831 07:59:23.567772 31878 rpcz_store.cc:276] 0831 07:59:22.279933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.319300 (+1039367us) service_pool.cc:225] Handling call
0831 07:59:23.319595 (+ 295us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:23.319797 (+ 202us) raft_consensus.cc:1532] Early marking committed up to index 5
0831 07:59:23.320108 (+ 311us) log.cc:844] Serialized 14 byte log entry
0831 07:59:23.563353 (+243245us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:23.563429 (+ 76us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:23.563491 (+ 62us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.563544 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:23.563654 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6149,"prepare.queue_time_us":239280,"thread_start_us":236750,"threads_started":1}
W20250831 07:59:23.569953 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:23.567013 31734 raft_consensus.cc:738] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:23.537916 1662 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:23.592373 31775 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:23.600303 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:23.608538 31699 leader_election.cc:336] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.768s (SENT)
I20250831 07:59:23.553406 1796 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:23.558933 1401 raft_consensus.cc:2747] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:23.590982 31734 consensus_queue.cc:260] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:23.611737 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 53226.2us
I20250831 07:59:23.613454 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:23.558085 940 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:23.516211 1525 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.626154 31725 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:23.630666 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 220) took 1078 ms. Trace:
I20250831 07:59:23.631052 31725 rpcz_store.cc:276] 0831 07:59:22.552589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.613052 (+1060463us) service_pool.cc:225] Handling call
0831 07:59:23.630589 (+ 17537us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:23.610272 32464 raft_consensus.cc:2391] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 6: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:23.636323 31849 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:23.636950 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 97035us
W20250831 07:59:23.640133 32436 leader_election.cc:336] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.153s (SENT)
I20250831 07:59:23.556071 1731 consensus_queue.cc:1035] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.061s
I20250831 07:59:23.640616 32436 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:23.642926 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:23.643086 31734 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:23.647639 31881 raft_consensus.cc:1273] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20250831 07:59:23.614985 1796 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:23.651948 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:23.653146 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1296) took 1541 ms. Trace:
I20250831 07:59:23.653602 31881 rpcz_store.cc:276] 0831 07:59:22.111625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:22.139594 (+ 27969us) service_pool.cc:225] Handling call
0831 07:59:23.645081 (+1505487us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:23.645256 (+ 175us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:23.648037 (+ 2781us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.653089 (+ 5052us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3588480}
I20250831 07:59:23.628798 1498 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:23.653363 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 136281us
I20250831 07:59:23.652861 31878 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
W20250831 07:59:23.663092 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 54062.1us
W20250831 07:59:23.659092 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 50687.3us
I20250831 07:59:23.625319 1812 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:23.643707 1813 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:23.668783 31773 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:23.648847 1401 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:23.644553 1737 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:23.570220 1666 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.072s
I20250831 07:59:23.677517 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:23.647864 1735 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:23.682930 31881 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 5
W20250831 07:59:23.681433 31698 leader_election.cc:336] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.947s (SENT)
I20250831 07:59:23.681787 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:23.662760 1606 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:23.683542 31698 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:23.660815 1275 consensus_queue.cc:260] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:23.687767 1814 log.cc:927] Time spent T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20250831 07:59:23.688031 1817 raft_consensus.cc:2802] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250831 07:59:23.688990 1815 log.cc:927] Time spent T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20250831 07:59:23.667631 1798 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:23.692386 1810 log.cc:927] Time spent T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.220s user 0.000s sys 0.001s
I20250831 07:59:23.678848 1813 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.679775 1401 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:23.693104 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 52999.1us
I20250831 07:59:23.693457 1821 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:23.693634 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:23.769292 31809 raft_consensus.cc:2391] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:23.693827 1809 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:23.694005 1683 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:23.778067 1683 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:23.778566 1683 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.778702 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:23.779665 31733 raft_consensus.cc:3053] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:23.780015 31733 raft_consensus.cc:738] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:23.780275 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:23.780740 1683 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:23.781291 1683 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:23.781808 1683 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.675535 1812 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.669786 1826 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:23.782824 1826 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:23.783846 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:23.784480 31726 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:23.785312 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:23.785506 1826 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:23.694828 1808 log.cc:927] Time spent T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
I20250831 07:59:23.786386 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:23.695324 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 219) took 1182 ms. Trace:
I20250831 07:59:23.683243 1577 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:23.788062 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:23.788237 1577 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:23.788651 1577 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:23.790107 1787 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.664806 1385 raft_consensus.cc:2747] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:23.792297 1385 consensus_queue.cc:1035] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250831 07:59:23.699410 1625 consensus_meta.cc:287] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.757s user 0.007s sys 0.000s
I20250831 07:59:23.788072 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1584) took 1865 ms. Trace:
I20250831 07:59:23.793232 1625 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:23.794166 32469 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:23.787276 31730 rpcz_store.cc:276] 0831 07:59:22.512672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.334436 (+821764us) service_pool.cc:225] Handling call
0831 07:59:23.334861 (+ 425us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:23.358756 (+ 23895us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:23.358785 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:23.368440 (+ 9655us) log.cc:844] Serialized 26 byte log entry
0831 07:59:23.538854 (+170414us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:23.558723 (+ 19869us) log.cc:844] Serialized 14 byte log entry
0831 07:59:23.558988 (+ 265us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.559060 (+ 72us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:23.695125 (+136065us) raft_consensus.cc:1683] finished
0831 07:59:23.695144 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:23.695264 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38024,"prepare.queue_time_us":24353,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":40,"raft.queue_time_us":52890,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":28561,"spinlock_wait_cycles":42547584,"thread_start_us":170067,"threads_started":1,"wal-append.queue_time_us":266636}
I20250831 07:59:23.700873 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:23.701604 1367 consensus_meta.cc:287] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.611s user 0.008s sys 0.000s
I20250831 07:59:23.795872 32462 raft_consensus.cc:1238] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:23.783859 1812 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:23.796365 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:23.796310 1367 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.796990 1803 raft_consensus.cc:2747] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:23.784303 1837 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:23.715561 1662 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.798496 1367 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:23.683084 1804 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:23.702291 1600 consensus_meta.cc:287] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.045s user 0.007s sys 0.003s
W20250831 07:59:23.707784 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:23.716078 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 43672.5us
I20250831 07:59:23.799866 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1303) took 1156 ms. Trace:
I20250831 07:59:23.671321 1686 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:23.800227 31775 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:23.800642 1686 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.801640 31775 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:23.659725 1669 consensus_queue.cc:1035] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.024s
I20250831 07:59:23.683346 1737 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.732123 1596 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:23.734140 1796 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:23.736193 1811 log.cc:927] Time spent T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.157s user 0.005s sys 0.005s
I20250831 07:59:23.725610 1817 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:23.692627 1735 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.759754 31734 raft_consensus.cc:1273] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20250831 07:59:23.761200 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 98613.5us
I20250831 07:59:23.824473 31698 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:23.760936 1740 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:23.763953 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:23.825531 31805 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 8.
I20250831 07:59:23.669767 1498 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.829072 1498 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:23.735721 1797 consensus_meta.cc:287] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.540s user 0.004s sys 0.003s
I20250831 07:59:23.829861 1797 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:23.832580 1797 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:59:23.833271 1797 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:23.833875 1797 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:23.777407 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:23.836264 31809 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:23.839545 31800 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:23.781886 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:23.841873 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:23.843223 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:23.785794 1772 log.cc:927] Time spent T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 1.968s user 0.018s sys 0.003s
W20250831 07:59:23.694208 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 57941.3us
I20250831 07:59:23.846328 31849 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:23.847146 1600 raft_consensus.cc:2747] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:23.795064 1625 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:23.683261 1816 consensus_queue.cc:1035] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20250831 07:59:23.848356 1367 consensus_queue.cc:1046] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250831 07:59:23.849376 1367 raft_consensus.cc:3053] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:23.849692 1367 raft_consensus.cc:738] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:23.849836 1816 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:23.850350 1816 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:23.850855 1816 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:23.850545 1367 consensus_queue.cc:260] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:23.780751 31733 consensus_queue.cc:260] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.852020 1367 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:23.852233 31733 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:23.799081 1804 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:23.799336 1662 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:23.776253 1275 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:23.776311 1018 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:23.800522 1809 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.729509 1525 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:23.729975 1396 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:23.802276 31801 raft_consensus.cc:1238] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:23.802426 31878 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:23.788655 32471 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 7
I20250831 07:59:23.800215 31875 rpcz_store.cc:276] 0831 07:59:22.643623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.441794 (+798171us) service_pool.cc:225] Handling call
0831 07:59:23.490097 (+ 48303us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:23.517758 (+ 27661us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:23.517802 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:23.518371 (+ 569us) log.cc:844] Serialized 26 byte log entry
0831 07:59:23.661046 (+142675us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:23.661088 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.661151 (+ 63us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:23.799670 (+138519us) raft_consensus.cc:1683] finished
0831 07:59:23.799687 (+ 17us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:23.799800 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13494,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":31282,"thread_start_us":89529,"threads_started":1,"wal-append.queue_time_us":219592}
I20250831 07:59:23.803712 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:23.799337 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1300) took 1630 ms. Trace:
I20250831 07:59:23.806408 31726 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:23.806895 31803 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:23.810716 31880 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 2.
I20250831 07:59:23.793702 31807 rpcz_store.cc:276] 0831 07:59:21.922414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.932081 (+ 9667us) service_pool.cc:225] Handling call
0831 07:59:21.932407 (+ 326us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:22.476878 (+544471us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:22.476908 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:22.515538 (+ 38630us) log.cc:844] Serialized 26 byte log entry
0831 07:59:23.256059 (+740521us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:23.281192 (+ 25133us) log.cc:844] Serialized 14 byte log entry
0831 07:59:23.281442 (+ 250us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.281536 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:23.787888 (+506352us) raft_consensus.cc:1683] finished
0831 07:59:23.787909 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:23.788020 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":27370,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":5247,"raft.queue_time_us":16574,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":184,"thread_start_us":995276,"threads_started":2,"wal-append.queue_time_us":841613}
I20250831 07:59:23.797072 31808 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:23.799165 1683 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:23.768180 1401 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:23.821499 31628 catalog_manager.cc:5582] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 3 to 5, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 5 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:23.762926 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:23.876922 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 52548.5us
I20250831 07:59:23.869767 31883 rpcz_store.cc:276] 0831 07:59:22.169119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:22.205959 (+ 36840us) service_pool.cc:225] Handling call
0831 07:59:22.567393 (+361434us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:22.567590 (+ 197us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:22.568004 (+ 414us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:22.568030 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:22.568547 (+ 517us) log.cc:844] Serialized 26 byte log entry
0831 07:59:23.129491 (+560944us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:23.129544 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.129620 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:23.799036 (+669416us) raft_consensus.cc:1683] finished
0831 07:59:23.799057 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:23.799279 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4165,"raft.queue_time_us":3546,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":26236,"spinlock_wait_cycles":40876928,"thread_start_us":560542,"threads_started":1,"wal-append.queue_time_us":756596}
I20250831 07:59:23.878003 31698 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:23.874471 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 72906.3us
I20250831 07:59:23.880829 31806 raft_consensus.cc:1238] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:23.857687 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:23.858343 31732 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:23.860620 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:23.855940 1523 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20250831 07:59:23.866807 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1307) took 1221 ms. Trace:
I20250831 07:59:23.870079 31730 raft_consensus.cc:2408] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 07:59:23.803970 1731 consensus_queue.cc:1035] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.085s
W20250831 07:59:23.885911 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 120008us
I20250831 07:59:23.885051 31876 rpcz_store.cc:276] 0831 07:59:22.645093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.513600 (+868507us) service_pool.cc:225] Handling call
0831 07:59:23.544946 (+ 31346us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:23.545252 (+ 306us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:23.545290 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:23.545903 (+ 613us) log.cc:844] Serialized 26 byte log entry
0831 07:59:23.627361 (+ 81458us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:23.627402 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.627481 (+ 79us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:23.866486 (+239005us) raft_consensus.cc:1683] finished
0831 07:59:23.866522 (+ 36us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:23.866725 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36096,"thread_start_us":81104,"threads_started":1,"wal-append.queue_time_us":313480}
I20250831 07:59:23.887790 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:23.891036 31730 raft_consensus.cc:3053] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:23.891651 31730 raft_consensus.cc:738] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:23.890395 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:23.896701 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:23.896502 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:23.860766 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:23.856190 1817 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:23.897094 31875 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:23.898129 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:23.905570 31725 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:23.868151 1826 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:23.875253 1662 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:23.903362 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:23.855616 1804 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.871935 1784 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:23.881866 1686 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:23.876103 1525 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:23.884709 1625 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:23.814739 1821 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.889225 1837 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250831 07:59:23.916922 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:23.887463 1666 consensus_queue.cc:1035] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
W20250831 07:59:23.922731 31729 consensus_meta.cc:287] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.519s user 0.006s sys 0.000s
I20250831 07:59:23.907465 31730 consensus_queue.cc:260] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.874954 1018 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:23.950685 1018 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:23.906867 31876 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:23.868221 1740 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:23.924067 1686 consensus_queue.cc:1035] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:59:23.910298 1662 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:23.953967 1662 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:23.916945 1826 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:23.925201 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:23.960318 31883 raft_consensus.cc:2391] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:23.961284 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:23.961702 1826 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:23.925406 1525 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:23.962451 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:23.963516 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:23.913892 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1301) took 1744 ms. Trace:
I20250831 07:59:23.964455 31879 rpcz_store.cc:276] 0831 07:59:22.169566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:22.169771 (+ 205us) service_pool.cc:225] Handling call
0831 07:59:22.569779 (+400008us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5800204398. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:22.569970 (+ 191us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:22.570360 (+ 390us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:22.570391 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:22.570883 (+ 492us) log.cc:844] Serialized 26 byte log entry
0831 07:59:23.095092 (+524209us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:23.095147 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:23.095228 (+ 81us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:23.901648 (+806420us) raft_consensus.cc:1683] finished
0831 07:59:23.901681 (+ 33us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:23.913799 (+ 12118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8771,"raft.queue_time_us":1181,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":13620,"spinlock_wait_cycles":40120192,"thread_start_us":523807,"threads_started":1,"wal-append.queue_time_us":868768}
I20250831 07:59:23.965360 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:23.966354 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:23.967283 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:23.918182 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:23.969002 31882 raft_consensus.cc:1238] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:23.971230 31882 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 07:59:23.969023 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:23.972865 31883 raft_consensus.cc:1238] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:23.908397 1787 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:23.946848 31730 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:23.967756 31879 raft_consensus.cc:1273] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:23.947345 31729 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
W20250831 07:59:23.938613 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:23.980461 31725 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
W20250831 07:59:23.984148 31877 consensus_meta.cc:287] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.513s user 0.009s sys 0.000s
I20250831 07:59:23.984723 31877 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
W20250831 07:59:23.985101 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:23.986100 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:23.986433 31881 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
W20250831 07:59:23.986960 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:23.987608 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:23.986356 31732 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:23.989336 31881 raft_consensus.cc:1238] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:23.945955 1797 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:24.011482 31847 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:24.005312 31875 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:24.014086 32436 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:23.963933 1525 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:24.022341 31882 raft_consensus.cc:1273] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250831 07:59:24.021077 31775 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:24.031128 31698 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:24.037755 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:24.011950 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:24.039461 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:23.927030 1784 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:24.050782 31775 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:24.050719 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:24.052280 31732 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:24.053915 31876 raft_consensus.cc:604] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 9 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:24.064831 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:24.065665 31732 raft_consensus.cc:2391] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 6: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:24.065407 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:24.074798 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 216851us
I20250831 07:59:24.075635 31882 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: []
W20250831 07:59:24.076431 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:24.037390 1837 raft_consensus.cc:2802] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:24.045590 1740 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.045941 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:24.090371 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:24.045461 1787 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:24.102609 31849 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:24.104127 1846 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:24.104988 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:24.105362 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:24.106283 31882 raft_consensus.cc:2408] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 07:59:24.088721 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:24.090764 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:24.110321 31732 raft_consensus.cc:2391] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 6: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:24.095850 1669 raft_consensus.cc:2747] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:24.096781 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 92607us
I20250831 07:59:24.097208 31773 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:24.028928 1797 consensus_queue.cc:1035] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20250831 07:59:24.112474 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:24.113551 31730 raft_consensus.cc:1273] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 07:59:24.114523 1853 consensus_queue.cc:1035] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.025s
I20250831 07:59:24.049633 1625 consensus_queue.cc:1046] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
W20250831 07:59:24.090382 1585 consensus_meta.cc:287] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.688s user 0.007s sys 0.004s
I20250831 07:59:24.126236 31773 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:24.133203 31849 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:24.136631 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:24.136036 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 85391.7us
I20250831 07:59:24.125196 1787 raft_consensus.cc:695] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:24.141824 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:24.142463 31879 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:24.142305 1787 consensus_queue.cc:237] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:24.136291 1438 raft_consensus.cc:2747] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:24.144547 1853 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250831 07:59:24.144598 1508 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250831 07:59:24.145525 31698 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:24.104969 1846 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:24.141126 31775 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:24.104460 1850 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:24.108415 1837 raft_consensus.cc:695] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:59:24.150365 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:24.109015 1796 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.109135 1737 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.150105 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:24.123425 1735 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.124624 1840 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:24.122884 1826 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:24.099292 1525 consensus_queue.cc:1046] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20250831 07:59:24.216989 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:24.158865 1669 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
W20250831 07:59:24.276077 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 184221us
I20250831 07:59:24.279783 31803 raft_consensus.cc:1238] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:24.279801 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:24.149938 1585 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:24.297183 31848 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:24.221131 1840 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:24.299250 31874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1681) took 2804 ms (client timeout 1947 ms). Trace:
I20250831 07:59:24.294546 32434 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:24.323927 31874 rpcz_store.cc:269] 0831 07:59:21.495244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.495395 (+ 151us) service_pool.cc:225] Handling call
0831 07:59:24.298987 (+2803592us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400057b54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:24.299153 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2659988736}
I20250831 07:59:24.168215 1662 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:24.346204 31734 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:24.167454 1809 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.149436 1846 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:24.299134 1796 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:24.298592 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:24.241214 1826 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:24.324491 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:24.218152 1850 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:24.297282 1804 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.322614 1275 raft_consensus.cc:3053] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:24.133368 1813 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.349750 31805 raft_consensus.cc:1238] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:24.365787 31874 raft_consensus.cc:1238] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:24.367365 32463 raft_consensus.cc:1238] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:24.324115 1603 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20250831 07:59:24.340994 1625 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:24.293232 1669 raft_consensus.cc:695] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:24.325470 1840 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:24.353468 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:24.368813 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:24.421015 31733 consensus_meta.cc:287] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.568s user 0.011s sys 0.000s
I20250831 07:59:24.392640 31807 raft_consensus.cc:1238] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
W20250831 07:59:24.426499 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 129413us
I20250831 07:59:24.426849 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:24.480746 31806 raft_consensus.cc:2408] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20250831 07:59:24.379606 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:24.324570 1683 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:24.460315 31728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1771) took 2453 ms (client timeout 2116 ms). Trace:
I20250831 07:59:24.425519 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1768) took 1129 ms. Trace:
I20250831 07:59:24.355967 1525 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:24.428853 1624 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 07:59:24.427753 1796 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:24.457316 1275 raft_consensus.cc:738] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:59:24.399230 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:24.423112 1859 raft_consensus.cc:2802] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:24.393289 1711 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.115s
W20250831 07:59:24.464576 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 312180us
I20250831 07:59:24.484522 31800 rpcz_store.cc:276] 0831 07:59:23.300620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.305074 (+ 4454us) service_pool.cc:225] Handling call
0831 07:59:23.305391 (+ 317us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:23.840100 (+534709us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:24.423607 (+583507us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b54001d8b54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:24.425457 (+ 1850us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1227813248}
W20250831 07:59:24.484459 31728 rpcz_store.cc:269] 0831 07:59:21.937834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.941105 (+ 3271us) service_pool.cc:225] Handling call
0831 07:59:24.391339 (+2450234us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540038dbd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:24.391511 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2426910976}
W20250831 07:59:24.428550 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 287496us
W20250831 07:59:24.492367 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:24.495942 31733 raft_consensus.cc:1273] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250831 07:59:24.547957 31733 pending_rounds.cc:85] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 3
I20250831 07:59:24.548309 31733 pending_rounds.cc:107] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250831 07:59:24.548647 31733 raft_consensus.cc:2887] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:24.423791 1861 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:24.515123 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:24.481886 31848 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:24.621623 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 139838us
I20250831 07:59:24.622395 31848 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:24.424161 1855 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.250s
I20250831 07:59:24.541507 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:24.531618 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:24.543192 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:24.423935 1835 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250831 07:59:24.603983 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 57261.4us
I20250831 07:59:24.601413 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:24.423887 1865 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250831 07:59:24.563714 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:24.604837 31775 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:24.613314 31874 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250831 07:59:24.498402 32469 raft_consensus.cc:1811] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Handling vote request from an unknown peer a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:24.623286 31848 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:24.479146 32471 consensus_meta.cc:287] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.595s user 0.007s sys 0.001s
I20250831 07:59:24.652855 32471 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:24.508980 1624 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:24.629320 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:24.518512 1796 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:24.540830 1821 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.543514 32463 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38366 (request call id 456) took 1533 ms. Trace:
I20250831 07:59:24.637658 31800 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:24.544790 1859 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:24.485363 1625 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:24.653955 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 520827us
I20250831 07:59:24.507086 1683 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:24.513093 1525 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:24.632864 1538 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250831 07:59:24.671983 32436 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.011s sys 0.068s
W20250831 07:59:24.648705 1849 log.cc:927] Time spent T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.290s user 0.001s sys 0.000s
I20250831 07:59:24.637495 1835 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:24.715952 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 92763.8us
I20250831 07:59:24.414484 1837 consensus_queue.cc:237] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:24.716259 31733 raft_consensus.cc:1273] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20250831 07:59:24.635258 1791 raft_consensus.cc:2802] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:24.633746 1861 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:24.709281 32436 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.590s user 0.011s sys 0.068s
W20250831 07:59:24.725447 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 605987us
W20250831 07:59:24.710011 1870 log.cc:927] Time spent T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.215s user 0.000s sys 0.000s
W20250831 07:59:24.713204 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 51609.5us
I20250831 07:59:24.635464 1844 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:24.708985 1525 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.727388 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:24.675616 1866 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:24.662014 1624 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:24.719465 1835 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:24.724745 1817 consensus_meta.cc:287] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.808s user 0.007s sys 0.004s
I20250831 07:59:24.764158 1817 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:24.766196 1817 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.727995 1662 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:24.731724 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:24.721710 32469 raft_consensus.cc:2391] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:24.711277 1826 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.709182 1538 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:24.781863 1662 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:24.782130 1538 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:24.782518 1662 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:24.783421 32464 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38366 (request call id 455) took 3165 ms (client timeout 2909 ms). Trace:
W20250831 07:59:24.783878 32464 rpcz_store.cc:269] 0831 07:59:21.618081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:21.629660 (+ 11579us) service_pool.cc:225] Handling call
0831 07:59:23.394296 (+1764636us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e5ce 000000000035a3c9 00007fdcecee5300 00007fdcecfabec4 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:24.783182 (+1388886us) spinlock_profiling.cc:243] Waited 84.3 ms on lock 0x7b540062f054. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:24.783346 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":204457600}
I20250831 07:59:24.784682 1662 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:24.784806 1802 raft_consensus.cc:3053] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:24.785290 1802 raft_consensus.cc:738] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:24.752741 1577 consensus_meta.cc:287] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.964s user 0.012s sys 0.000s
W20250831 07:59:24.721051 1842 log.cc:927] Time spent T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20250831 07:59:24.734319 1874 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:24.710840 1850 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.741307 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:24.741902 31881 raft_consensus.cc:1238] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20250831 07:59:24.705971 1859 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:24.707509 1879 raft_consensus.cc:2747] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:24.750749 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:24.810000 1740 raft_consensus.cc:3053] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:24.810635 1740 raft_consensus.cc:738] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:24.810989 1861 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:24.811748 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:24.706331 31849 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:24.812848 1686 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250831 07:59:24.812887 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 106635us
W20250831 07:59:24.814654 1401 consensus_meta.cc:287] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.911s user 0.008s sys 0.000s
I20250831 07:59:24.755362 1875 consensus_queue.cc:1046] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:59:24.814934 31849 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:24.756551 1683 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:24.815042 1401 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:24.738974 1872 consensus_queue.cc:1046] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20250831 07:59:24.815522 1686 raft_consensus.cc:3053] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:24.815937 1686 raft_consensus.cc:738] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:24.816358 1872 raft_consensus.cc:3053] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:24.816707 1872 raft_consensus.cc:738] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:24.816911 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:24.816736 1686 consensus_queue.cc:260] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:24.817600 1872 consensus_queue.cc:260] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:24.818837 1872 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:24.819510 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:50826 (request call id 19) took 1095 ms. Trace:
I20250831 07:59:24.819907 31876 rpcz_store.cc:276] 0831 07:59:23.736927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.889912 (+152985us) service_pool.cc:225] Handling call
0831 07:59:24.819345 (+929433us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b540020c6d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ff7813 00007fdcecf5d1ba 00007fdce65f71c4 00007fdce65f7152 00007fdce65f7081 00007fdce65f6ffc 00007fdce65f62b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:24.819429 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":235806976}
I20250831 07:59:24.730373 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:24.768323 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:24.779371 31882 raft_consensus.cc:2391] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
W20250831 07:59:24.753738 1862 log.cc:927] Time spent T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.413s user 0.001s sys 0.000s
I20250831 07:59:24.800771 1866 raft_consensus.cc:695] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:24.662060 1275 consensus_queue.cc:260] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:24.722167 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:24.776120 1817 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:24.792402 31628 catalog_manager.cc:5582] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 2 to 3, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 3 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:24.793201 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:24.797166 1625 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.715361 1669 consensus_queue.cc:237] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:24.745939 1844 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:24.661850 1596 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:24.799691 1835 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:24.809022 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:24.748788 1264 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20250831 07:59:24.810117 1791 raft_consensus.cc:695] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:24.801839 1874 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:24.818091 1686 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:24.758138 1869 log.cc:927] Time spent T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.157s user 0.001s sys 0.001s
I20250831 07:59:24.710194 1603 consensus_queue.cc:1046] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20250831 07:59:24.823339 32436 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:24.754741 31800 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:24.828470 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1796) took 1056 ms. Trace:
I20250831 07:59:24.709483 1585 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:24.829135 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1780) took 1244 ms. Trace:
I20250831 07:59:24.820350 31773 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:24.864637 31801 rpcz_store.cc:276] 0831 07:59:23.585177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.585464 (+ 287us) service_pool.cc:225] Handling call
0831 07:59:23.615662 (+ 30198us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:23.868706 (+253044us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:24.828873 (+960167us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b54001da454. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:24.829052 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":428671872}
I20250831 07:59:24.834982 31877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 236) took 1111 ms. Trace:
I20250831 07:59:24.865783 31877 rpcz_store.cc:276] 0831 07:59:23.716790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.987162 (+270372us) service_pool.cc:225] Handling call
0831 07:59:23.987518 (+ 356us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:23.987975 (+ 457us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:23.988005 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:24.016979 (+ 28974us) log.cc:844] Serialized 26 byte log entry
0831 07:59:24.154253 (+137274us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:24.154309 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:24.154403 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:24.828230 (+673827us) raft_consensus.cc:1683] finished
0831 07:59:24.828252 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:24.828393 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17938,"raft.queue_time_us":112,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":18204,"thread_start_us":105313,"threads_started":1,"wal-append.queue_time_us":218525}
I20250831 07:59:24.832854 31733 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:24.869113 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:24.869917 31733 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:24.870972 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:24.872622 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 49404us
I20250831 07:59:24.841547 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:24.809531 1525 consensus_queue.cc:1046] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250831 07:59:24.885239 31732 raft_consensus.cc:2391] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 7: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:24.891201 31847 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:24.925064 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:24.845276 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:24.853055 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 65153.7us
I20250831 07:59:24.853732 31849 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:24.837046 1275 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:24.674937 32463 rpcz_store.cc:276] 0831 07:59:23.001993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.259718 (+257725us) service_pool.cc:225] Handling call
0831 07:59:24.366944 (+1107226us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:24.367160 (+ 216us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:24.463483 (+ 96323us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:24.534826 (+ 71343us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9632000}
W20250831 07:59:24.857507 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 421184us
I20250831 07:59:24.948779 32435 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:24.858913 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:24.859211 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:24.786180 1577 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:24.862442 31848 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:24.862226 31726 rpcz_store.cc:276] 0831 07:59:23.771781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.786128 (+ 14347us) service_pool.cc:225] Handling call
0831 07:59:24.828279 (+1042151us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b5400310e54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:24.828401 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44604672}
I20250831 07:59:24.953075 31698 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:24.835502 1855 consensus_queue.cc:1035] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
W20250831 07:59:24.962044 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 70965.1us
W20250831 07:59:24.963649 31880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1297) took 2846 ms (client timeout 1923 ms). Trace:
W20250831 07:59:24.837957 1303 consensus_queue.cc:1162] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8247a38b550649aa9cd6e7668dcaec10" responder_term: 6 status { last_received { term: 3 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: [5.3-5.3]" } } last_received_current_leader { term: 0 index: 0 } }
W20250831 07:59:24.965792 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 145553us
W20250831 07:59:24.964118 31880 rpcz_store.cc:269] 0831 07:59:22.117610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:22.140832 (+ 23222us) service_pool.cc:225] Handling call
0831 07:59:24.963288 (+2822456us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400410854. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:24.963467 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2970336000}
I20250831 07:59:24.956106 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:24.961707 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:24.972532 31733 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:24.837005 1817 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:24.964974 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:24.973454 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:24.965322 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 102972us
I20250831 07:59:24.965838 31698 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:24.974494 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:24.975071 31733 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:24.976080 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:24.976174 31698 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:24.981925 32471 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250831 07:59:24.861035 1874 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:24.948199 1861 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:24.955052 1525 consensus_queue.cc:1230] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20250831 07:59:25.011102 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 47652.7us
I20250831 07:59:24.985627 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:25.015017 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:24.987057 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 133398us
I20250831 07:59:25.018777 31726 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:24.972307 1880 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:25.020149 31849 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:25.030012 1881 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:25.030604 1881 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:25.031183 1817 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:25.032855 1881 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:25.033607 1882 raft_consensus.cc:2747] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:25.016964 1385 raft_consensus.cc:2747] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:25.036677 1784 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:25.037336 1840 consensus_meta.cc:287] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.518s user 0.007s sys 0.000s
I20250831 07:59:25.037775 1840 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:25.038363 32462 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38366 (request call id 459) took 1599 ms. Trace:
I20250831 07:59:25.038789 32462 rpcz_store.cc:276] 0831 07:59:23.439358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.443736 (+ 4378us) service_pool.cc:225] Handling call
0831 07:59:23.795655 (+351919us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:23.796388 (+ 733us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:25.038157 (+1241769us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b54004c13d0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:25.038305 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":630700288}
I20250831 07:59:25.018098 1861 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:25.040123 1861 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:25.040302 1791 consensus_queue.cc:237] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:25.043851 31806 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:24.972878 1860 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:24.961611 1884 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:25.010192 1875 raft_consensus.cc:2802] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:25.051730 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:25.060937 31800 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:24.987874 1837 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 1 FOLLOWER]: Advancing to term 2
I20250831 07:59:24.973503 1813 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:24.982954 1850 raft_consensus.cc:2747] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250831 07:59:25.017100 1018 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.043s sys 0.071s
I20250831 07:59:25.043118 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:24.986459 1711 consensus_queue.cc:1046] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
W20250831 07:59:25.139194 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:25.114215 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:25.078889 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:25.120257 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:25.086537 31883 raft_consensus.cc:1238] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
W20250831 07:59:25.146261 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 127107us
I20250831 07:59:25.081894 31726 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:25.148934 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:25.149749 32470 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
I20250831 07:59:25.149958 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:25.151108 31847 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:25.152070 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1706) took 1582 ms. Trace:
I20250831 07:59:25.152351 31726 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:25.153656 31883 rpcz_store.cc:276] 0831 07:59:23.569219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:25.085895 (+1516676us) service_pool.cc:225] Handling call
0831 07:59:25.086308 (+ 413us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:25.151714 (+ 65406us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:25.152002 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:25.156021 31733 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:25.157181 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:25.047237 1840 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:25.163511 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:25.061189 1882 raft_consensus.cc:2802] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:24.861195 1866 consensus_queue.cc:237] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:25.092842 1816 consensus_meta.cc:287] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 1.241s user 0.010s sys 0.004s
I20250831 07:59:25.171267 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:25.101608 1841 log.cc:927] Time spent T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.761s user 0.006s sys 0.000s
I20250831 07:59:25.049551 1826 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:25.176985 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:25.145278 1018 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.731s user 0.044s sys 0.077s
I20250831 07:59:25.021983 1683 consensus_queue.cc:1046] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
I20250831 07:59:25.061532 1787 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.113s
I20250831 07:59:25.072572 1600 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:25.150894 1821 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:25.175634 31729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 208) took 2965 ms (client timeout 1924 ms). Trace:
I20250831 07:59:25.175982 31731 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20250831 07:59:25.177302 31800 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:25.184700 1882 raft_consensus.cc:695] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:25.187373 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:25.101476 1844 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:25.162866 1875 raft_consensus.cc:695] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:25.188509 1600 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:25.028169 1880 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:25.077195 1888 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:25.181860 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:25.089082 1835 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:25.079751 1577 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:25.085574 1860 raft_consensus.cc:695] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:25.101949 1401 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:25.100092 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:25.175323 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:25.210390 31726 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
W20250831 07:59:25.185871 31729 rpcz_store.cc:269] 0831 07:59:22.209901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:22.556644 (+346743us) service_pool.cc:225] Handling call
0831 07:59:25.175437 (+2618793us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b54003e1f54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:25.175566 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1852046208}
I20250831 07:59:25.188851 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:25.215261 31800 raft_consensus.cc:2408] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 07:59:25.193374 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:25.193903 31881 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:25.192571 31801 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:25.193392 31730 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d"
dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
mode: GRACEFUL
new_leader_uuid: "8247a38b550649aa9cd6e7668dcaec10"
from {username='slave'} at 127.0.0.1:48724
I20250831 07:59:25.217141 31730 raft_consensus.cc:604] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Received request to transfer leadership to TS 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:25.217486 31730 raft_consensus.cc:612] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:59:25.207545 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:25.172067 1731 raft_consensus.cc:2802] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:25.058774 1498 consensus_queue.cc:1046] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.124s
I20250831 07:59:25.190829 1600 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:25.085364 1538 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:25.077445 1796 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:25.213666 32436 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:25.219873 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:25.219837 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:25.220340 31733 raft_consensus.cc:2391] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 7: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:25.068334 1385 consensus_queue.cc:1046] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.168s
I20250831 07:59:25.253679 31732 raft_consensus.cc:1238] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
W20250831 07:59:25.228446 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 202469us
I20250831 07:59:25.222280 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:25.285030 31806 raft_consensus.cc:2391] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:25.262930 31803 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
W20250831 07:59:25.323776 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:25.348165 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:25.346298 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:25.221697 1880 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:25.194202 1826 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:25.191481 1816 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:25.191960 1859 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:25.189667 1875 consensus_queue.cc:237] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:25.197242 1821 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:25.202248 1813 consensus_queue.cc:1046] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.199s
W20250831 07:59:25.630414 31876 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.018s sys 0.021s
W20250831 07:59:26.145045 31876 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.067s user 0.026s sys 0.033s
W20250831 07:59:25.886288 31847 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.021s sys 0.029s
W20250831 07:59:26.146519 31847 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.960s user 0.023s sys 0.036s
W20250831 07:59:26.146999 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 961156us
W20250831 07:59:25.920878 31877 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.013s sys 0.037s
W20250831 07:59:26.148154 31877 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.076s user 0.022s sys 0.037s
W20250831 07:59:26.148841 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:25.809898 31801 consensus_meta.cc:287] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.593s user 0.010s sys 0.000s
W20250831 07:59:26.043058 31699 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.008s sys 0.005s
I20250831 07:59:25.542344 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:25.451642 31874 raft_consensus.cc:1238] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20250831 07:59:25.733596 31881 consensus_meta.cc:287] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.512s user 0.009s sys 0.001s
W20250831 07:59:25.913158 31698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.025s sys 0.033s
I20250831 07:59:25.219386 1731 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:25.633345 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:25.444319 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.154875 31800 raft_consensus.cc:2408] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250831 07:59:25.402036 31808 raft_consensus.cc:1238] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
W20250831 07:59:25.401067 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 187480us
W20250831 07:59:26.148907 31773 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.004s sys 0.004s
W20250831 07:59:26.157359 31773 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.664s user 0.004s sys 0.004s
W20250831 07:59:26.157711 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 664985us
I20250831 07:59:26.145093 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:25.355057 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.151413 31881 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:26.160795 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1735) took 1011 ms. Trace:
I20250831 07:59:26.161206 31881 rpcz_store.cc:276] 0831 07:59:25.148941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:25.166672 (+ 17731us) service_pool.cc:225] Handling call
0831 07:59:26.160722 (+994050us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:26.145170 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 1.12502e+06us
W20250831 07:59:26.151844 31698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.996s user 0.027s sys 0.039s
W20250831 07:59:26.163712 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 1.00845e+06us
W20250831 07:59:26.150652 31699 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.883s user 0.010s sys 0.006s
W20250831 07:59:26.167250 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 899956us
W20250831 07:59:26.168756 31699 leader_election.cc:336] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.508s (SENT)
I20250831 07:59:26.169353 31699 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:26.151741 31801 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:26.146337 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 221) took 2743 ms. Trace:
I20250831 07:59:26.174019 31731 rpcz_store.cc:276] 0831 07:59:23.402748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.404709 (+ 1961us) service_pool.cc:225] Handling call
0831 07:59:25.175619 (+1770910us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:25.186434 (+ 10815us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:26.146041 (+959607us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b54003e1f50. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:26.146263 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3968244096}
I20250831 07:59:26.160197 31803 raft_consensus.cc:2408] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250831 07:59:25.289136 1803 raft_consensus.cc:2802] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 07:59:26.085906 1275 consensus_meta.cc:287] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 1.124s user 0.009s sys 0.000s
W20250831 07:59:26.131292 1872 consensus_meta.cc:287] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.312s user 0.009s sys 0.002s
I20250831 07:59:26.212234 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:26.125387 1686 consensus_meta.cc:287] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.245s user 0.009s sys 0.001s
I20250831 07:59:26.176261 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1612) took 1071 ms. Trace:
I20250831 07:59:26.223474 31879 raft_consensus.cc:1238] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:26.223413 31806 rpcz_store.cc:276] 0831 07:59:25.105276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:25.433479 (+328203us) service_pool.cc:225] Handling call
0831 07:59:26.176167 (+742688us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.165855 1385 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:26.229067 31728 raft_consensus.cc:2408] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20250831 07:59:25.947712 1837 consensus_meta.cc:287] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.786s user 0.003s sys 0.009s
I20250831 07:59:26.094946 1538 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:26.201128 31802 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:25.467427 1879 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:26.101006 1796 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:26.185921 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1613) took 1017 ms. Trace:
I20250831 07:59:26.225327 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 235) took 1076 ms. Trace:
I20250831 07:59:26.187386 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1603) took 1353 ms. Trace:
W20250831 07:59:26.011111 1864 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.025s sys 0.049s
W20250831 07:59:26.043643 1887 log.cc:927] Time spent T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.450s user 0.001s sys 0.000s
W20250831 07:59:26.049487 1757 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.853s user 0.002s sys 0.008s
I20250831 07:59:25.886165 1840 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:25.964773 1889 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.162569 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:25.791378 1895 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.155876 1731 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:26.232003 1385 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.233940 1821 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:26.194939 1803 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:26.195166 1525 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.234869 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:26.302337 1757 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.100s user 0.002s sys 0.010s
I20250831 07:59:26.228329 1401 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:26.236061 1793 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.236516 1835 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.238775 1508 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.238729 1498 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:26.328692 1498 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:26.330417 1385 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:26.330673 1498 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:26.332113 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 241) took 1577 ms. Trace:
I20250831 07:59:26.332494 31874 rpcz_store.cc:276] 0831 07:59:24.768280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:24.780131 (+ 11851us) service_pool.cc:225] Handling call
0831 07:59:24.792612 (+ 12481us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:26.151217 (+1358605us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:26.331847 (+180630us) spinlock_profiling.cc:243] Waited 659 ms on lock 0x7b5400246b54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:26.332052 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1514906368}
I20250831 07:59:26.239347 1275 raft_consensus.cc:2747] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:26.239794 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1782) took 2471 ms. Trace:
W20250831 07:59:26.243316 1885 log.cc:927] Time spent T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 1.265s user 0.000s sys 0.050s
I20250831 07:59:26.246017 1495 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:26.248078 1740 consensus_queue.cc:260] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.352500 1740 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:26.254352 1850 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:25.210871 1624 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.353657 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1317) took 2294 ms. Trace:
I20250831 07:59:26.354084 31875 rpcz_store.cc:276] 0831 07:59:24.059323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:24.122809 (+ 63486us) service_pool.cc:225] Handling call
0831 07:59:24.123256 (+ 447us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:24.341603 (+218347us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:24.341650 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:24.425111 (+ 83461us) log.cc:844] Serialized 26 byte log entry
0831 07:59:24.724231 (+299120us) raft_consensus.cc:1643] Marking committed up to 1
0831 07:59:24.724256 (+ 25us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:24.724302 (+ 46us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:26.242635 (+1518333us) spinlock_profiling.cc:243] Waited 54.2 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:26.353376 (+110741us) raft_consensus.cc:1683] finished
0831 07:59:26.353406 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:26.353598 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8392,"raft.queue_time_us":36821,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":6772,"spinlock_wait_cycles":124701568,"thread_start_us":403904,"threads_started":2,"wal-append.queue_time_us":364066}
I20250831 07:59:26.355321 1624 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:26.355052 1802 consensus_queue.cc:260] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.278785 31731 rpcz_store.cc:276] 0831 07:59:25.148448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.191692 (+1043244us) service_pool.cc:225] Handling call
0831 07:59:26.224821 (+ 33129us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:26.224993 (+ 172us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:26.225022 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:26.225066 (+ 44us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:26.225102 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:26.225151 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:26.225275 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.278980 31803 rpcz_store.cc:276] 0831 07:59:24.834119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:25.354744 (+520625us) service_pool.cc:225] Handling call
0831 07:59:26.187313 (+832569us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.408191 31874 raft_consensus.cc:1273] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:26.408811 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.278719 31800 rpcz_store.cc:276] 0831 07:59:25.168987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:25.271355 (+102368us) service_pool.cc:225] Handling call
0831 07:59:26.185836 (+914481us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.277408 1837 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 07:59:26.269210 1874 consensus_meta.cc:287] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.231s user 0.010s sys 0.002s
I20250831 07:59:26.311666 31881 raft_consensus.cc:2391] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
W20250831 07:59:26.284237 1864 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.200s user 0.027s sys 0.049s
I20250831 07:59:26.235776 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:25.287977 1882 consensus_queue.cc:237] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:26.321429 31806 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 2.
W20250831 07:59:26.316975 1791 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.095s user 0.052s sys 0.077s
I20250831 07:59:26.321368 1803 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:26.335579 31734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1800) took 2562 ms (client timeout 2026 ms). Trace:
I20250831 07:59:26.326357 1901 raft_consensus.cc:3053] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:26.318601 1896 consensus_queue.cc:1035] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20250831 07:59:26.452622 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.453454 31800 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:26.454103 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1621) took 1263 ms. Trace:
I20250831 07:59:26.454495 31800 rpcz_store.cc:276] 0831 07:59:25.190888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.409953 (+1219065us) service_pool.cc:225] Handling call
0831 07:59:26.454033 (+ 44080us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.322613 1889 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:25.593104 1860 consensus_queue.cc:237] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.315418 1840 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:26.340369 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 244) took 1571 ms. Trace:
I20250831 07:59:26.344528 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 298) took 2080 ms. Trace:
I20250831 07:59:26.334936 31727 rpcz_store.cc:276] 0831 07:59:23.768292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.768404 (+ 112us) service_pool.cc:225] Handling call
0831 07:59:23.778245 (+ 9841us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:23.901005 (+122760us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:23.901049 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:23.901136 (+ 87us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:23.901217 (+ 81us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:26.239536 (+2338319us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b540013fe54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:26.239582 (+ 46us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:26.239723 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":246831360}
W20250831 07:59:26.168423 1861 consensus_meta.cc:287] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 1.128s user 0.013s sys 0.007s
I20250831 07:59:26.356205 31882 raft_consensus.cc:1238] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20250831 07:59:26.358083 31725 raft_consensus.cc:1273] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:26.408842 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1331) took 1007 ms. Trace:
I20250831 07:59:26.335049 1793 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.416139 31803 raft_consensus.cc:2391] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:26.466981 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1619) took 1276 ms. Trace:
I20250831 07:59:26.462000 31876 rpcz_store.cc:276] 0831 07:59:24.769017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:24.952477 (+183460us) service_pool.cc:225] Handling call
0831 07:59:24.973422 (+ 20945us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:24.973637 (+ 215us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:24.973676 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:25.020236 (+ 46560us) log.cc:844] Serialized 26 byte log entry
0831 07:59:26.145802 (+1125566us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:26.145840 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:26.145905 (+ 65us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:26.340060 (+194155us) raft_consensus.cc:1683] finished
0831 07:59:26.340083 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:26.340279 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1067980,"threads_started":1,"wal-append.queue_time_us":1266793}
W20250831 07:59:26.447554 31734 rpcz_store.cc:269] 0831 07:59:23.772818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.859667 (+ 86849us) service_pool.cc:225] Handling call
0831 07:59:26.335331 (+2475664us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b54000203d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:26.335487 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":770694400}
I20250831 07:59:26.468004 1793 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:26.462116 31804 rpcz_store.cc:276] 0831 07:59:24.263722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:24.278996 (+ 15274us) service_pool.cc:225] Handling call
0831 07:59:24.321717 (+ 42721us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:24.644522 (+322805us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:24.644571 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:24.645151 (+ 580us) log.cc:844] Serialized 26 byte log entry
0831 07:59:24.862476 (+217325us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:24.862515 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:24.862576 (+ 61us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:26.344209 (+1481633us) raft_consensus.cc:1683] finished
0831 07:59:26.344238 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:26.344436 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17361,"raft.queue_time_us":85690,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":24927,"thread_start_us":429519,"threads_started":2,"wal-append.queue_time_us":536648}
I20250831 07:59:26.469398 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.470517 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.471148 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1626) took 1025 ms. Trace:
I20250831 07:59:26.471441 31804 rpcz_store.cc:276] 0831 07:59:25.445454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.470288 (+1024834us) service_pool.cc:225] Handling call
0831 07:59:26.471095 (+ 807us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.448870 1802 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:26.448427 31729 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.909s user 0.031s sys 0.060s
I20250831 07:59:26.303179 1879 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:26.327230 1401 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:26.327196 1525 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:26.449113 1791 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.230s user 0.052s sys 0.078s
I20250831 07:59:26.496865 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.466068 31874 rpcz_store.cc:276] 0831 07:59:25.401820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.333717 (+931897us) service_pool.cc:225] Handling call
0831 07:59:26.407863 (+ 74146us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:26.408603 (+ 740us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:26.408789 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.438405 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1802) took 1085 ms. Trace:
I20250831 07:59:26.497946 31806 rpcz_store.cc:276] 0831 07:59:25.352869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.226200 (+873331us) service_pool.cc:225] Handling call
0831 07:59:26.438325 (+212125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.451143 1901 raft_consensus.cc:738] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:59:26.472200 31729 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.933s user 0.031s sys 0.060s
I20250831 07:59:26.335108 1835 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.353669 1850 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:26.472944 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.335685 1508 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.498927 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.467418 31803 rpcz_store.cc:276] 0831 07:59:25.190388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.408593 (+1218205us) service_pool.cc:225] Handling call
0831 07:59:26.466926 (+ 58333us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.501189 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1625) took 1280 ms. Trace:
I20250831 07:59:26.516156 31804 rpcz_store.cc:276] 0831 07:59:25.220494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.472011 (+1251517us) service_pool.cc:225] Handling call
0831 07:59:26.501100 (+ 29089us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.515763 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.518059 31803 raft_consensus.cc:2391] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:26.518520 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1606) took 1575 ms. Trace:
I20250831 07:59:26.518872 31803 rpcz_store.cc:276] 0831 07:59:24.943458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.515434 (+1571976us) service_pool.cc:225] Handling call
0831 07:59:26.518454 (+ 3020us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:26.461738 1866 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.025s sys 0.070s
I20250831 07:59:26.504164 31800 raft_consensus.cc:2391] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:26.320013 1895 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.520619 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.528030 31803 raft_consensus.cc:2391] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:26.526124 31806 raft_consensus.cc:2391] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:26.528898 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1623) took 1337 ms. Trace:
I20250831 07:59:26.529245 31806 rpcz_store.cc:276] 0831 07:59:25.191405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.498650 (+1307245us) service_pool.cc:225] Handling call
0831 07:59:26.528842 (+ 30192us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:26.529795 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 366089us
I20250831 07:59:26.436683 1874 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.539011 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.535245 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:26.526429 1866 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s user 0.028s sys 0.070s
I20250831 07:59:26.550606 31849 leader_election.cc:400] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Vote denied by peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027) with higher term. Message: Invalid argument: T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5.
I20250831 07:59:26.548573 31803 raft_consensus.cc:2408] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 07:59:26.539860 31804 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:26.551014 31849 leader_election.cc:403] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250831 07:59:26.550652 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.552390 31800 raft_consensus.cc:2391] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:26.550814 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.553961 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.473901 1861 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.554700 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1607) took 1610 ms. Trace:
I20250831 07:59:26.556375 31803 rpcz_store.cc:276] 0831 07:59:24.943881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.553721 (+1609840us) service_pool.cc:225] Handling call
0831 07:59:26.554630 (+ 909us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.513816 1859 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:26.557425 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.527180 1791 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:26.559258 31875 raft_consensus.cc:1238] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:26.564939 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.568763 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.569794 31803 raft_consensus.cc:2391] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:59:26.554848 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:26.571978 1901 consensus_queue.cc:260] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:26.573396 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1618) took 1383 ms. Trace:
I20250831 07:59:26.567301 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:26.578720 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:26.574196 31803 rpcz_store.cc:276] 0831 07:59:25.190129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.568341 (+1378212us) service_pool.cc:225] Handling call
0831 07:59:26.573315 (+ 4974us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.619552 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:26.615746 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 446978us
I20250831 07:59:26.620193 31806 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:26.574398 31800 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 2.
I20250831 07:59:26.620460 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:26.622416 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.642427 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43428 (request call id 312) took 1587 ms. Trace:
W20250831 07:59:26.620575 31699 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.843s (SENT) [suppressed 36 similar messages]
I20250831 07:59:26.646119 31800 rpcz_store.cc:276] 0831 07:59:25.054770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.553009 (+1498239us) service_pool.cc:225] Handling call
0831 07:59:26.642347 (+ 89338us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:26.646677 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:26.644227 31803 raft_consensus.cc:2391] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:59:26.640288 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:26.648983 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:26.618260 1879 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:26.654688 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:26.658604 31699 leader_election.cc:336] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.528s (SENT)
I20250831 07:59:26.553910 1867 consensus_queue.cc:1035] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.082s
I20250831 07:59:26.558672 1826 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:26.723476 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:26.664378 31698 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:26.725562 31727 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:26.664351 32434 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.011s sys 0.015s
W20250831 07:59:26.726493 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 67940.3us
W20250831 07:59:26.727077 32434 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s user 0.013s sys 0.021s
W20250831 07:59:26.728719 31699 leader_election.cc:336] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.820s (SENT)
W20250831 07:59:26.730183 31699 leader_election.cc:336] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.926s (SENT)
I20250831 07:59:26.730670 31699 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:26.665038 31881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "a0d845d01c8c4894895f66e3beae298c"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:26.597630 1793 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.733604 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:26.618956 1837 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.757956 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:26.621846 1827 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:26.761035 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 230889us
I20250831 07:59:26.761716 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:26.619123 1275 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:26.726164 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 61878.9us
W20250831 07:59:26.794643 31699 leader_election.cc:336] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.938s (SENT)
W20250831 07:59:26.763582 31878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1312) took 3094 ms (client timeout 1823 ms). Trace:
I20250831 07:59:26.766651 31734 pending_rounds.cc:85] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 2
I20250831 07:59:26.634157 1668 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.573941 1790 raft_consensus.cc:2747] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027) with higher term. Message: Invalid argument: T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5.
I20250831 07:59:26.749902 31881 raft_consensus.cc:604] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Received request to transfer leadership to TS a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:26.769897 31849 connection.cc:232] RPC call timeout handler was delayed by -1.2877938730000551s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2026s for a response
I20250831 07:59:26.573225 1791 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:26.790699 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:26.792483 31800 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:26.664613 1821 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.643167 1625 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.759835 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.798575 31881 raft_consensus.cc:612] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:59:26.795933 31734 pending_rounds.cc:107] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250831 07:59:26.795258 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:26.795784 31878 rpcz_store.cc:269] 0831 07:59:23.668952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.684420 (+ 15468us) service_pool.cc:225] Handling call
0831 07:59:26.761045 (+3076625us) spinlock_profiling.cc:243] Waited 6.3 ms on lock 0x7b54001ed2d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:26.763470 (+ 2425us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14487936}
I20250831 07:59:26.795140 31699 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:26.633715 1859 consensus_queue.cc:1035] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.174s
I20250831 07:59:26.800307 31698 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:26.793234 31775 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:26.813567 31734 raft_consensus.cc:2887] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:26.799889 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.828866 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1617) took 1639 ms. Trace:
I20250831 07:59:26.829262 31804 rpcz_store.cc:276] 0831 07:59:25.189869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.734759 (+1544890us) service_pool.cc:225] Handling call
0831 07:59:26.828799 (+ 94040us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.830261 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.831053 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1616) took 1641 ms. Trace:
I20250831 07:59:26.831372 31804 rpcz_store.cc:276] 0831 07:59:25.189578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.829959 (+1640381us) service_pool.cc:225] Handling call
0831 07:59:26.831001 (+ 1042us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:26.825803 31699 leader_election.cc:336] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.993s (SENT)
I20250831 07:59:26.767884 1826 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:26.839202 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:26.790879 1843 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.834082 31874 raft_consensus.cc:1273] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 07:59:26.834244 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:26.843515 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:26.836613 31699 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:26.838336 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:26.792307 1879 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.770965 1757 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:26.825259 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:26.794994 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:26.843062 32436 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:26.851166 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:26.795225 1910 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:26.854733 1910 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.856880 1910 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:26.858719 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:26.859339 31804 raft_consensus.cc:1811] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Handling vote request from an unknown peer a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:26.859915 31804 raft_consensus.cc:2391] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 2: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:26.864077 31881 raft_consensus.cc:1238] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20250831 07:59:26.768500 1793 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:26.869701 31803 raft_consensus.cc:1273] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
W20250831 07:59:26.869805 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 69606.8us
I20250831 07:59:26.838624 1840 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:26.873337 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1611) took 1825 ms. Trace:
I20250831 07:59:26.878793 31804 rpcz_store.cc:276] 0831 07:59:25.048007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.836277 (+1788270us) service_pool.cc:225] Handling call
0831 07:59:26.873279 (+ 37002us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.873186 31698 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:26.877204 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:26.876613 31800 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:26.877959 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:26.870139 31803 pending_rounds.cc:85] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 5
I20250831 07:59:26.880302 31881 raft_consensus.cc:2408] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250831 07:59:26.880517 31803 pending_rounds.cc:107] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20250831 07:59:26.880829 31803 raft_consensus.cc:2887] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:26.881822 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 309) took 2590 ms. Trace:
I20250831 07:59:26.882422 31800 rpcz_store.cc:276] 0831 07:59:24.291033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.880359 (+2589326us) service_pool.cc:225] Handling call
0831 07:59:26.880637 (+ 278us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:26.881373 (+ 736us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:26.881412 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:26.881471 (+ 59us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:26.881531 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:26.881609 (+ 78us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:26.881749 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.797190 1827 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
W20250831 07:59:26.863166 1495 consensus_meta.cc:287] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.510s user 0.005s sys 0.005s
I20250831 07:59:26.858634 1879 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:26.858770 1757 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.864718 1816 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:26.865262 1525 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:26.801798 1275 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:26.853061 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:26.808152 1877 consensus_queue.cc:1035] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20250831 07:59:26.942140 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 306) took 2652 ms. Trace:
I20250831 07:59:26.942539 31803 rpcz_store.cc:276] 0831 07:59:24.290126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.839211 (+2549085us) service_pool.cc:225] Handling call
0831 07:59:26.863463 (+ 24252us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:26.934495 (+ 71032us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:26.942085 (+ 7590us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:26.798225 1837 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:26.812112 1668 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:26.949836 31806 raft_consensus.cc:2408] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250831 07:59:27.056411 31727 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:26.833273 1821 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:27.057579 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:26.838609 1625 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:27.072729 1920 raft_consensus.cc:2747] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:27.073168 1909 log.cc:927] Time spent T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.261s user 0.000s sys 0.000s
I20250831 07:59:27.073591 1793 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:27.074921 1904 log.cc:927] Time spent T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.507s user 0.000s sys 0.001s
W20250831 07:59:27.076494 1916 log.cc:927] Time spent T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.239s user 0.002s sys 0.000s
I20250831 07:59:26.933961 1850 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:27.078290 1803 consensus_meta.cc:287] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.617s user 0.005s sys 0.002s
I20250831 07:59:27.078682 1803 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:27.079352 1625 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:27.081218 1861 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:27.081455 1853 consensus_queue.cc:1046] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250831 07:59:27.082715 1853 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:27.082799 1861 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250831 07:59:26.943457 1835 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:27.084856 1874 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:27.077713 1802 consensus_meta.cc:287] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.599s user 0.009s sys 0.000s
W20250831 07:59:27.099730 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:26.852311 1843 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:27.033468 1898 log.cc:927] Time spent T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.532s user 0.001s sys 0.000s
I20250831 07:59:27.034029 1508 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:27.068825 1525 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:27.102895 1275 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:27.103837 1908 log.cc:927] Time spent T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.332s user 0.000s sys 0.001s
I20250831 07:59:27.057274 1669 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:26.885602 1922 raft_consensus.cc:2747] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:27.104090 1757 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:27.059146 1840 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:27.086165 1826 raft_consensus.cc:2747] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:27.106467 1827 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:26.941447 1737 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:27.106910 1939 raft_consensus.cc:3053] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:27.107133 1837 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:26.878139 1401 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:27.340018 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 234) took 2191 ms. Trace:
I20250831 07:59:26.941524 1303 raft_consensus.cc:2747] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:27.353515 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1313) took 2591 ms. Trace:
I20250831 07:59:27.354084 31726 rpcz_store.cc:276] 0831 07:59:24.761805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:25.211036 (+449231us) service_pool.cc:225] Handling call
0831 07:59:25.268993 (+ 57957us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 07:59:25.346296 (+ 77303us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:25.346322 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 07:59:25.346943 (+ 621us) log.cc:844] Serialized 52 byte log entry
0831 07:59:26.234608 (+887665us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b18002dfd00. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce507e0f7 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdce7c3b2bb 00007fdce7c40dc3 00007fdce7c41171 00007fdce7fd009f
0831 07:59:26.619612 (+385004us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:26.826437 (+206825us) log.cc:844] Serialized 14 byte log entry
0831 07:59:27.341263 (+514826us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:27.341348 (+ 85us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:27.341394 (+ 46us) raft_consensus.cc:1683] finished
0831 07:59:27.353233 (+ 11839us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:27.353441 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":679072,"prepare.queue_time_us":459940,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":306778,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":103419,"spinlock_wait_cycles":7611648,"thread_start_us":853623,"threads_started":3,"wal-append.queue_time_us":1735063}
W20250831 07:59:27.343429 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 470305us
W20250831 07:59:27.341615 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 548453us
I20250831 07:59:27.358286 31775 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:27.343117 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:27.341001 31730 rpcz_store.cc:276] 0831 07:59:25.148186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:25.345740 (+197554us) service_pool.cc:225] Handling call
0831 07:59:25.346076 (+ 336us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:26.232353 (+886277us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:26.232387 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:26.232890 (+ 503us) log.cc:844] Serialized 26 byte log entry
0831 07:59:26.535383 (+302493us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:26.811177 (+275794us) log.cc:844] Serialized 14 byte log entry
0831 07:59:26.811403 (+ 226us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:26.811475 (+ 72us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:27.339750 (+528275us) raft_consensus.cc:1683] finished
0831 07:59:27.339778 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:27.339953 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":915198,"prepare.queue_time_us":543676,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":954048,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":60725,"thread_start_us":550112,"threads_started":2,"wal-append.queue_time_us":603682}
I20250831 07:59:27.343189 31881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
W20250831 07:59:27.350991 31732 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.004s sys 0.007s
W20250831 07:59:27.351397 31773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.007s sys 0.002s
W20250831 07:59:27.365492 31732 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.747s user 0.004s sys 0.008s
I20250831 07:59:27.344630 1840 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:27.351611 1821 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:27.671036 31881 raft_consensus.cc:604] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
W20250831 07:59:27.463917 32435 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.004s sys 0.012s
I20250831 07:59:27.446381 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:27.627367 32436 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.015s sys 0.008s
W20250831 07:59:27.673000 32436 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.815s user 0.015s sys 0.010s
I20250831 07:59:27.672961 31730 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
W20250831 07:59:27.673545 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 815342us
I20250831 07:59:27.673987 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1324) took 1016 ms. Trace:
I20250831 07:59:27.674443 31730 rpcz_store.cc:276] 0831 07:59:26.657120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.360194 (+703074us) service_pool.cc:225] Handling call
0831 07:59:27.673858 (+313664us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:27.417304 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 59893.7us
W20250831 07:59:27.277443 1740 consensus_meta.cc:287] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.924s user 0.011s sys 0.000s
W20250831 07:59:27.676292 31773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.847s user 0.018s sys 0.008s
W20250831 07:59:27.445808 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 87638.5us
I20250831 07:59:27.417881 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:27.684617 31726 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:27.672199 32435 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.736s user 0.006s sys 0.022s
I20250831 07:59:27.616911 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1788) took 3886 ms. Trace:
I20250831 07:59:27.686823 31805 rpcz_store.cc:276] 0831 07:59:23.730390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.826330 (+ 95940us) service_pool.cc:225] Handling call
0831 07:59:23.826712 (+ 382us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:24.460743 (+634031us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:27.616539 (+3155796us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b54000ceed4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:27.616822 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1202457984}
I20250831 07:59:27.412967 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:27.136673 1731 consensus_meta.cc:287] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.813s user 0.009s sys 0.000s
W20250831 07:59:27.534698 31699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.012s sys 0.002s
W20250831 07:59:27.697384 31699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.852s user 0.015s sys 0.007s
I20250831 07:59:27.340597 1939 raft_consensus.cc:738] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:27.352934 1889 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:27.698021 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 872206us
I20250831 07:59:27.174710 1803 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:27.438848 1525 raft_consensus.cc:695] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:59:27.136147 1931 log.cc:927] Time spent T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20250831 07:59:27.342375 1791 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:27.685493 31775 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:27.712996 31775 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:27.411623 1943 raft_consensus.cc:2802] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:27.690173 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:27.722680 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1332) took 1008 ms. Trace:
I20250831 07:59:27.723100 31730 rpcz_store.cc:276] 0831 07:59:26.714352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.689009 (+974657us) service_pool.cc:225] Handling call
0831 07:59:27.722610 (+ 33601us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:27.724244 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:27.725059 31730 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:27.728832 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1323) took 1072 ms. Trace:
I20250831 07:59:27.729141 31730 rpcz_store.cc:276] 0831 07:59:26.656732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.723933 (+1067201us) service_pool.cc:225] Handling call
0831 07:59:27.728785 (+ 4852us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:27.729990 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:27.730518 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:27.730677 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1331) took 1016 ms. Trace:
I20250831 07:59:27.208035 1668 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:27.701252 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:27.701632 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:27.732515 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:27.534929 1875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.010s sys 0.032s
W20250831 07:59:27.733093 1875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.895s user 0.019s sys 0.032s
I20250831 07:59:27.733173 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:27.631098 1898 log.cc:927] Time spent T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.211s user 0.000s sys 0.000s
I20250831 07:59:27.670001 1840 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:27.690914 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:27.731003 31730 rpcz_store.cc:276] 0831 07:59:26.714008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.729754 (+1015746us) service_pool.cc:225] Handling call
0831 07:59:27.730632 (+ 878us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:27.103720 1895 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:27.284235 1877 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:27.692441 1498 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:27.699565 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1325) took 1042 ms. Trace:
I20250831 07:59:27.736423 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:27.738756 31734 rpcz_store.cc:276] 0831 07:59:26.657425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.353961 (+696536us) service_pool.cc:225] Handling call
0831 07:59:27.699504 (+345543us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:27.739362 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1330) took 1025 ms. Trace:
I20250831 07:59:27.352196 1737 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:27.708925 1731 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:27.737738 1740 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:27.740289 31730 rpcz_store.cc:276] 0831 07:59:26.713624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.736098 (+1022474us) service_pool.cc:225] Handling call
0831 07:59:27.739304 (+ 3206us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:27.757601 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 44718.2us
I20250831 07:59:27.757696 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:27.764146 31883 raft_consensus.cc:2391] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:27.764313 31775 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:27.387853 1669 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:27.753348 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:27.759711 1955 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:27.773236 1525 consensus_queue.cc:237] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:27.765442 31775 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:27.774467 1740 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:27.775288 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:27.775573 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:27.775888 1804 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:27.776338 1804 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:27.738021 1943 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:27.776809 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:27.789379 32435 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:27.765444 32467 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:27.758906 31726 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:27.769356 32471 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:27.739883 1401 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250831 07:59:27.773079 31834 heartbeater.cc:646] Failed to heartbeat to 127.29.217.62:40065 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.29.217.62:40065 timed out after 15.000s (SENT)
I20250831 07:59:27.773695 1955 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:27.716831 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:27.815388 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:27.800467 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1326) took 1142 ms. Trace:
I20250831 07:59:27.816964 31877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 242) took 3048 ms. Trace:
I20250831 07:59:27.801273 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:27.817618 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:27.817111 31726 rpcz_store.cc:276] 0831 07:59:26.657729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.356852 (+699123us) service_pool.cc:225] Handling call
0831 07:59:27.800407 (+443555us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:27.817414 31877 rpcz_store.cc:276] 0831 07:59:24.768584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:24.868063 (+ 99479us) service_pool.cc:225] Handling call
0831 07:59:24.963921 (+ 95858us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:25.016865 (+ 52944us) raft_consensus.cc:1532] Early marking committed up to index 6
0831 07:59:25.054476 (+ 37611us) log.cc:844] Serialized 14 byte log entry
0831 07:59:26.154915 (+1100439us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:26.179654 (+ 24739us) log.cc:844] Serialized 26 byte log entry
0831 07:59:26.179858 (+ 204us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:26.179909 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:26.180003 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:27.816741 (+1636738us) raft_consensus.cc:1683] finished
0831 07:59:27.816766 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:27.816899 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":112518,"prepare.queue_time_us":37396,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":62,"raft.queue_time_us":164000,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":32669,"spinlock_wait_cycles":39808,"thread_start_us":1075983,"threads_started":1,"wal-append.queue_time_us":1359440}
I20250831 07:59:27.819116 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:27.820127 1940 raft_consensus.cc:2802] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:27.787650 1852 raft_consensus.cc:2802] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:27.820561 1940 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:27.754513 1498 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:27.817252 1955 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:27.833134 1940 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:27.777027 32462 raft_consensus.cc:1238] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:27.834410 31730 raft_consensus.cc:2408] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250831 07:59:27.778270 1901 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:27.752719 1877 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:27.777388 1957 raft_consensus.cc:2747] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:27.799947 1884 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:27.771445 1853 consensus_meta.cc:287] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.682s user 0.012s sys 0.000s
I20250831 07:59:27.800134 1508 consensus_queue.cc:1046] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20250831 07:59:27.832857 1804 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:27.840097 1852 raft_consensus.cc:695] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:27.820992 1843 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:27.799983 1803 consensus_queue.cc:1046] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20250831 07:59:27.799012 1943 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:27.860252 31699 connection.cc:232] RPC call timeout handler was delayed by -1.0375656390000358s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33640000000000003s for a response
W20250831 07:59:27.887394 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 122020us
I20250831 07:59:27.893730 31775 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:27.797065 1939 consensus_queue.cc:260] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:27.841912 1946 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
W20250831 07:59:27.896562 31699 leader_election.cc:336] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 2.229s (SENT)
I20250831 07:59:27.900863 31699 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:27.902172 31699 leader_election.cc:336] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 2.465s (SENT)
I20250831 07:59:27.897033 31775 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:27.867326 1853 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 07:59:27.900164 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:27.868474 1957 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:27.907101 32470 raft_consensus.cc:2408] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250831 07:59:27.906868 1957 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:27.913966 1957 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:27.904233 1683 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:27.917346 1877 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:27.920662 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1736) took 1290 ms. Trace:
I20250831 07:59:27.921178 31874 rpcz_store.cc:276] 0831 07:59:26.630403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.632321 (+ 1918us) service_pool.cc:225] Handling call
0831 07:59:26.665481 (+ 33160us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:26.843974 (+178493us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:27.920341 (+1076367us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b5400410d54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:27.920585 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":386996480}
I20250831 07:59:27.899940 1843 raft_consensus.cc:2802] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:27.942286 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:27.949777 1840 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:27.917336 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:27.865408 1884 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:27.924037 1821 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:27.943634 1843 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:28.000582 1958 log.cc:927] Time spent T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250831 07:59:28.022667 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1792) took 3663 ms. Trace:
I20250831 07:59:28.023125 31803 rpcz_store.cc:276] 0831 07:59:24.359442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.341806 (+2982364us) service_pool.cc:225] Handling call
0831 07:59:27.414885 (+ 73079us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:27.415335 (+ 450us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 07:59:27.721632 (+306297us) log.cc:844] Serialized 14 byte log entry
0831 07:59:27.781218 (+ 59586us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:27.787362 (+ 6144us) log.cc:844] Serialized 26 byte log entry
0831 07:59:27.787538 (+ 176us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:27.787575 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:27.787642 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:28.022407 (+234765us) raft_consensus.cc:1683] finished
0831 07:59:28.022433 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.022594 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17578,"prepare.queue_time_us":386080,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":53974,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":21674,"thread_start_us":132543,"threads_started":2,"wal-append.queue_time_us":143306}
I20250831 07:59:28.026935 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1795) took 3605 ms. Trace:
I20250831 07:59:28.017462 32466 raft_consensus.cc:2391] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:28.032531 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38366 (request call id 460) took 3866 ms. Trace:
I20250831 07:59:28.032907 32466 rpcz_store.cc:276] 0831 07:59:24.166187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:24.216645 (+ 50458us) service_pool.cc:225] Handling call
0831 07:59:25.033772 (+817127us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e5ce 000000000035a3c9 00007fdcecee5300 00007fdcecfabec4 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:27.917220 (+2883448us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e66c 00007fdce505e5d6 000000000035a3c9 00007fdcecee5300 00007fdcecfabec4 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:28.032460 (+115240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51829888}
I20250831 07:59:27.986518 1731 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:28.035110 31698 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:28.036011 1683 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:28.036535 31803 rpcz_store.cc:276] 0831 07:59:24.421717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.025370 (+3603653us) service_pool.cc:225] Handling call
0831 07:59:28.025787 (+ 417us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.026007 (+ 220us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:28.026043 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:28.026142 (+ 99us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:28.026208 (+ 66us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.026279 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.026869 (+ 590us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:28.018229 1821 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:28.041705 1821 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:28.019573 1843 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:28.047248 1683 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:27.992007 1852 consensus_queue.cc:237] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:28.049240 1603 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:28.049948 1840 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:28.049876 1603 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:28.050215 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:28.052222 1603 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:28.076387 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:28.095737 31726 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
W20250831 07:59:28.115733 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 255629us
W20250831 07:59:28.118713 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:28.119371 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1328) took 1459 ms. Trace:
I20250831 07:59:28.121870 31734 rpcz_store.cc:276] 0831 07:59:26.659989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.739614 (+1079625us) service_pool.cc:225] Handling call
0831 07:59:27.762877 (+ 23263us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:27.763244 (+ 367us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:27.763269 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:27.787885 (+ 24616us) log.cc:844] Serialized 26 byte log entry
0831 07:59:27.897620 (+109735us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:27.897665 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:27.897884 (+ 219us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:28.119101 (+221217us) raft_consensus.cc:1683] finished
0831 07:59:28.119129 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.119301 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":533,"raft.queue_time_us":786,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":17908,"thread_start_us":77278,"threads_started":1,"wal-append.queue_time_us":303169}
I20250831 07:59:28.123898 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:28.124739 31734 raft_consensus.cc:2408] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 07:59:28.125550 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:28.126636 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:28.127650 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:28.129546 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:28.130486 31734 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:28.131534 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:28.141451 31734 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
I20250831 07:59:28.143428 32436 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:28.144649 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:28.146824 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:28.145512 31699 leader_election.cc:336] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 3.069s (SENT)
I20250831 07:59:28.147769 31699 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:28.149765 31734 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:28.151538 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:28.152695 31734 raft_consensus.cc:2408] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
W20250831 07:59:28.153429 1891 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 3.122s user 0.022s sys 0.046s
I20250831 07:59:28.154851 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:28.156036 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 308) took 3865 ms. Trace:
I20250831 07:59:28.156431 31734 raft_consensus.cc:3058] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:28.150282 1901 raft_consensus.cc:2747] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:28.156534 31804 rpcz_store.cc:276] 0831 07:59:24.290753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.879601 (+2588848us) service_pool.cc:225] Handling call
0831 07:59:27.061952 (+182351us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:27.062208 (+ 256us) raft_consensus.cc:1532] Early marking committed up to index 8
0831 07:59:27.726978 (+664770us) log.cc:844] Serialized 14 byte log entry
0831 07:59:28.155684 (+428706us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:28.155742 (+ 58us) raft_consensus.cc:1643] Marking committed up to 8
0831 07:59:28.155797 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.155881 (+ 84us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.155987 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11769,"prepare.queue_time_us":717721,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":32728,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":4565,"spinlock_wait_cycles":7722368,"thread_start_us":806641,"threads_started":3,"wal-append.queue_time_us":173105}
I20250831 07:59:28.168289 1943 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:28.156067 1891 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 3.125s user 0.022s sys 0.046s
I20250831 07:59:28.176911 1966 raft_consensus.cc:2802] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:28.183718 31734 raft_consensus.cc:2466] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:28.185269 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:28.186779 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:28.188298 31804 raft_consensus.cc:1238] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:28.188711 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:28.189498 31734 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:28.188887 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1797) took 3706 ms. Trace:
I20250831 07:59:28.192363 31804 rpcz_store.cc:276] 0831 07:59:24.483045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.178115 (+3695070us) service_pool.cc:225] Handling call
0831 07:59:28.188115 (+ 10000us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.188673 (+ 558us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.188839 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:28.194792 31803 raft_consensus.cc:1238] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:28.192031 1966 raft_consensus.cc:695] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [term 7 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:28.196460 1966 consensus_queue.cc:237] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:28.197291 1940 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:28.198117 31734 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:28.199193 32436 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:28.201296 31804 raft_consensus.cc:1273] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20250831 07:59:28.201624 1940 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:28.201906 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1791) took 4431 ms. Trace:
I20250831 07:59:28.202093 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1798) took 3718 ms. Trace:
I20250831 07:59:28.202500 31804 rpcz_store.cc:276] 0831 07:59:24.483391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.193442 (+3710051us) service_pool.cc:225] Handling call
0831 07:59:28.200969 (+ 7527us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.201755 (+ 786us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.202044 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:28.202431 31725 rpcz_store.cc:276] 0831 07:59:23.770760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:24.003687 (+232927us) service_pool.cc:225] Handling call
0831 07:59:26.336059 (+2332372us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:26.465781 (+129722us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.201723 (+1735942us) spinlock_profiling.cc:243] Waited 2.33 s on lock 0x7b54000203d0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:28.201850 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5362738048}
I20250831 07:59:28.180091 1843 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:28.204260 31804 raft_consensus.cc:1238] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:28.204802 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1799) took 3428 ms. Trace:
I20250831 07:59:28.205073 31804 rpcz_store.cc:276] 0831 07:59:24.776024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.203795 (+3427771us) service_pool.cc:225] Handling call
0831 07:59:28.204105 (+ 310us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.204594 (+ 489us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.204753 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:28.206311 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:28.207227 31734 raft_consensus.cc:2408] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250831 07:59:28.208779 31734 raft_consensus.cc:1273] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:28.209188 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:28.209344 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1329) took 1549 ms. Trace:
I20250831 07:59:28.209568 1943 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:28.209829 31725 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:28.210125 31734 rpcz_store.cc:276] 0831 07:59:26.660294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.207922 (+1547628us) service_pool.cc:225] Handling call
0831 07:59:28.208484 (+ 562us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.209119 (+ 635us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.209293 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:28.211352 31725 raft_consensus.cc:1238] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 6. Current term is 7. Ops: [5.6-6.7]
I20250831 07:59:28.211948 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1800) took 3435 ms. Trace:
I20250831 07:59:28.212334 31806 rpcz_store.cc:276] 0831 07:59:24.776397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.211225 (+3434828us) service_pool.cc:225] Handling call
0831 07:59:28.211481 (+ 256us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.211635 (+ 154us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:28.211658 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:28.211697 (+ 39us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:28.211726 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.211777 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.211899 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:28.216104 32436 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:28.175793 1955 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:28.221971 1949 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:28.222564 1949 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:28.222889 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39068 (request call id 638) took 4375 ms. Trace:
I20250831 07:59:28.222983 1949 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:28.223378 31809 rpcz_store.cc:276] 0831 07:59:23.847201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.853826 (+ 6625us) service_pool.cc:225] Handling call
0831 07:59:28.222320 (+4368494us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b54002f0654. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f81cd0 00007fdce8005eea 00007fdcecf5e60e 00007fdce65fbcc7 00007fdce65fbc52 00007fdce65fbb81 00007fdce65fbafc 00007fdce65fadb2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:28.222829 (+ 509us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5571305856}
I20250831 07:59:28.228690 31804 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:28.219792 1940 raft_consensus.cc:2802] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election won for term 6
W20250831 07:59:28.239893 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:28.239729 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1624) took 3047 ms. Trace:
I20250831 07:59:28.241079 31804 rpcz_store.cc:276] 0831 07:59:25.193971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.213881 (+3019910us) service_pool.cc:225] Handling call
0831 07:59:28.228454 (+ 14573us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.239259 (+ 10805us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.239673 (+ 414us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:28.226289 1843 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:28.253940 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 48842.2us
I20250831 07:59:28.243254 1940 raft_consensus.cc:695] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:28.266180 31880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1330) took 2919 ms. Trace:
W20250831 07:59:28.268545 31699 leader_election.cc:336] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.614s (SENT)
I20250831 07:59:28.269083 31699 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:28.266630 31880 rpcz_store.cc:276] 0831 07:59:25.361476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:25.361627 (+ 151us) service_pool.cc:225] Handling call
0831 07:59:25.572124 (+210497us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:25.572447 (+ 323us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 07:59:26.547973 (+975526us) log.cc:844] Serialized 14 byte log entry
0831 07:59:28.265615 (+1717642us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b1800140a40. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce501365a 00007fdcee4e4409 00007fdce4ed4af0 00007fdce5098894 00007fdce5098805 00007fdce7ff5020 00007fdce80076c6 00007fdce8018237 00007fdce80181b8
0831 07:59:28.265756 (+ 141us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:28.265807 (+ 51us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:28.265863 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.265957 (+ 94us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.266080 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":594838,"prepare.queue_time_us":1199546,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":6808,"spinlock_wait_cycles":34029824,"thread_start_us":576152,"threads_started":2,"wal-append.queue_time_us":387222}
W20250831 07:59:28.271700 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:28.276578 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 379598us
I20250831 07:59:28.277705 31775 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:28.278872 31849 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:28.282760 1911 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:28.284637 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1794) took 3924 ms. Trace:
I20250831 07:59:28.285686 1827 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:28.286233 1827 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:28.288007 1882 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:28.288740 1791 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:28.288874 1882 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:28.289947 1668 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:28.290306 1895 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:28.290674 1378 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:28.291157 1882 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:28.291543 1940 consensus_queue.cc:237] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:28.292672 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:28.291165 1378 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:28.293485 32466 raft_consensus.cc:2391] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:28.285020 31807 rpcz_store.cc:276] 0831 07:59:24.360110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.814826 (+3454716us) service_pool.cc:225] Handling call
0831 07:59:27.842704 (+ 27878us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:28.140059 (+297355us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:28.140104 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:28.140651 (+ 547us) log.cc:844] Serialized 26 byte log entry
0831 07:59:28.238256 (+ 97605us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:28.238297 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.238572 (+ 275us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:28.284421 (+ 45849us) raft_consensus.cc:1683] finished
0831 07:59:28.284441 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.284576 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":363,"raft.queue_time_us":48888,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":3625,"thread_start_us":358116,"threads_started":2,"wal-append.queue_time_us":97982}
I20250831 07:59:28.289255 1791 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:28.290840 1895 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:28.297752 31699 leader_election.cc:336] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.644s (SENT)
I20250831 07:59:28.297061 31849 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:28.343675 31847 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:28.348465 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 50749.2us
I20250831 07:59:28.306797 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:28.348903 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 51936.6us
W20250831 07:59:28.331661 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:28.349571 31849 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:28.350664 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:28.350775 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:28.298013 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:28.351267 31807 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:28.351651 31878 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:28.285161 1897 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:28.304944 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:28.305235 1950 consensus_queue.cc:1035] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:59:28.305632 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:28.349663 31847 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:28.305898 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:28.290417 1668 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:28.288295 1827 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:28.351135 31849 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
W20250831 07:59:28.358197 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:28.281296 1401 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:28.305655 1378 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:28.288116 1240 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:28.356362 1666 raft_consensus.cc:2747] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:28.280110 1735 raft_consensus.cc:2747] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:28.368896 31849 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:28.377628 31881 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:28.333011 1891 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:28.378268 31849 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:28.305658 1955 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:28.373971 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1636) took 2147 ms. Trace:
I20250831 07:59:28.341094 1882 raft_consensus.cc:2747] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:28.383766 31807 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 3.
W20250831 07:59:28.374675 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 97145.1us
I20250831 07:59:28.385279 31775 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:28.374063 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 310) took 4082 ms. Trace:
I20250831 07:59:28.371524 31698 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:28.374307 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1316) took 3611 ms. Trace:
I20250831 07:59:28.386754 31698 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:28.378401 31881 raft_consensus.cc:604] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:28.386217 31800 rpcz_store.cc:276] 0831 07:59:24.291351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.341790 (+3050439us) service_pool.cc:225] Handling call
0831 07:59:27.414086 (+ 72296us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:27.414354 (+ 268us) raft_consensus.cc:1532] Early marking committed up to index 7
0831 07:59:27.722330 (+307976us) log.cc:844] Serialized 14 byte log entry
0831 07:59:28.373747 (+651417us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:28.373798 (+ 51us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:28.373841 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.373899 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.374012 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":467458,"prepare.queue_time_us":385793,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":3297,"raft.queue_time_us":467935,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"thread_start_us":473475,"threads_started":2,"wal-append.queue_time_us":397528}
I20250831 07:59:28.368283 1538 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:28.367005 1668 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:28.359442 1787 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:28.390762 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:28.391530 31808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 300) took 4125 ms. Trace:
I20250831 07:59:28.283633 1911 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:28.391983 31808 rpcz_store.cc:276] 0831 07:59:24.265794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:24.265974 (+ 180us) service_pool.cc:225] Handling call
0831 07:59:24.293600 (+ 27626us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:26.168087 (+1874487us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.391274 (+2223187us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540006fc54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:28.391457 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2548758912}
I20250831 07:59:28.393558 1911 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:28.381166 1666 raft_consensus.cc:2747] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:28.394088 32435 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:28.372375 1949 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:28.397770 1949 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:28.358711 1897 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:28.399384 1401 raft_consensus.cc:695] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:28.382023 1859 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:28.400432 1826 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:28.400633 1859 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:28.402480 1859 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:28.403106 1950 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:28.403496 1950 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:28.404266 1950 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:28.386363 31729 rpcz_store.cc:276] 0831 07:59:24.762599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:25.212214 (+449615us) service_pool.cc:225] Handling call
0831 07:59:25.269650 (+ 57436us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:25.345868 (+ 76218us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 07:59:26.502113 (+1156245us) log.cc:844] Serialized 14 byte log entry
0831 07:59:28.197256 (+1695143us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b18002df8e0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b8f4 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:59:28.373712 (+176456us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b18002df8e0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce7c4955c 00007fdce507e77b 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdce7c3b2bb 00007fdce7c40dc3 00007fdce7c412da
0831 07:59:28.373962 (+ 250us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:28.374033 (+ 71us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:28.374086 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.374146 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.374257 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1296723,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":8341,"raft.queue_time_us":1369,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":69,"spinlock_wait_cycles":116031360,"thread_start_us":1178820,"threads_started":2,"wal-append.queue_time_us":555179}
W20250831 07:59:28.405958 31883 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 246) took 2825 ms (client timeout 3161 ms). Trace:
I20250831 07:59:28.393496 31808 raft_consensus.cc:1238] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 8. Current term is 9. Ops: []
I20250831 07:59:28.388309 31698 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:28.372870 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:28.407756 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1801) took 3620 ms. Trace:
I20250831 07:59:28.408197 31807 raft_consensus.cc:1273] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 07:59:28.368489 1378 raft_consensus.cc:2802] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:28.382845 31809 rpcz_store.cc:276] 0831 07:59:26.226176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.225021 (+1998845us) service_pool.cc:225] Handling call
0831 07:59:28.225472 (+ 451us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:28.225707 (+ 235us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:28.225744 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:28.226340 (+ 596us) log.cc:844] Serialized 26 byte log entry
0831 07:59:28.252145 (+ 25805us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:28.255256 (+ 3111us) log.cc:844] Serialized 14 byte log entry
0831 07:59:28.373688 (+118432us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.373755 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:28.373791 (+ 36us) raft_consensus.cc:1683] finished
0831 07:59:28.373804 (+ 13us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.373915 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":118178,"prepare.queue_time_us":3684,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":118679,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":76,"thread_start_us":11815,"threads_started":1,"wal-append.queue_time_us":44822}
I20250831 07:59:28.374958 1804 raft_consensus.cc:2747] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:28.442853 1378 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:28.371179 1910 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:28.443847 1538 raft_consensus.cc:2747] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:28.444108 1843 consensus_queue.cc:1046] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:28.444934 31809 raft_consensus.cc:1238] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:28.443861 31775 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:28.477927 1966 consensus_queue.cc:1046] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250831 07:59:28.478646 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:28.479350 31725 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:28.479101 1804 raft_consensus.cc:2802] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:28.480222 31804 raft_consensus.cc:1273] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 07:59:28.481547 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1638) took 2195 ms. Trace:
I20250831 07:59:28.480235 1804 raft_consensus.cc:695] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:28.481875 31804 rpcz_store.cc:276] 0831 07:59:26.285933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.242125 (+1956192us) service_pool.cc:225] Handling call
0831 07:59:28.479909 (+237784us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.480630 (+ 721us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.481377 (+ 747us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b5400029750. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:28.481496 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":526139264}
I20250831 07:59:28.483409 31804 raft_consensus.cc:1238] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:28.483124 1804 consensus_queue.cc:237] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:28.446693 31800 raft_consensus.cc:1238] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:28.446759 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 243) took 3678 ms. Trace:
I20250831 07:59:28.477923 31775 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:28.485374 31800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 313) took 1927 ms. Trace:
I20250831 07:59:28.485716 31775 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:28.485812 31800 rpcz_store.cc:276] 0831 07:59:26.558210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.390187 (+1831977us) service_pool.cc:225] Handling call
0831 07:59:28.446506 (+ 56319us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.484842 (+ 38336us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.485313 (+ 471us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:28.447027 31808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 314) took 1879 ms. Trace:
I20250831 07:59:28.446224 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1805) took 2144 ms. Trace:
I20250831 07:59:28.448355 1949 consensus_queue.cc:1046] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250831 07:59:28.369414 1791 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:28.443208 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 65001.1us
I20250831 07:59:28.409098 1840 raft_consensus.cc:2747] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:28.400432 1401 consensus_queue.cc:237] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:28.418012 1897 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:28.489760 1962 raft_consensus.cc:2747] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:28.498430 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:28.498668 1932 log.cc:927] Time spent T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 1.414s user 0.009s sys 0.012s
I20250831 07:59:28.486079 1877 raft_consensus.cc:2802] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:28.499289 1877 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:28.451444 1735 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:28.460915 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:28.501205 31726 raft_consensus.cc:1238] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
W20250831 07:59:28.470062 1917 log.cc:927] Time spent T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 1.389s user 0.000s sys 0.016s
I20250831 07:59:28.461004 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:28.471222 1843 raft_consensus.cc:3053] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:28.471608 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1327) took 3701 ms. Trace:
I20250831 07:59:28.451692 1378 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:28.460772 1946 raft_consensus.cc:3053] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:28.448945 1757 raft_consensus.cc:2802] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:28.461422 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250831 07:59:28.387001 1891 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:28.371754 1240 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:28.559415 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 245) took 3013 ms. Trace:
I20250831 07:59:28.561062 31875 rpcz_store.cc:276] 0831 07:59:25.545776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.378084 (+832308us) service_pool.cc:225] Handling call
0831 07:59:26.378443 (+ 359us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:26.565018 (+186575us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.559145 (+1994127us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b540001d6d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:28.559350 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":415264000}
W20250831 07:59:28.560236 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 74598.2us
I20250831 07:59:28.562647 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 299) took 4297 ms. Trace:
I20250831 07:59:28.563299 31775 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:28.473245 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:28.462713 32464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47886 (request call id 441) took 1820 ms. Trace:
W20250831 07:59:28.465551 1928 log.cc:927] Time spent T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 1.358s user 0.016s sys 0.013s
I20250831 07:59:28.468770 1895 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:28.478735 31801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1605) took 3618 ms (client timeout 1644 ms). Trace:
W20250831 07:59:28.565779 31801 rpcz_store.cc:269] 0831 07:59:24.860596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:24.951853 (+ 91257us) service_pool.cc:225] Handling call
0831 07:59:28.478545 (+3526692us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b5400029754. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:28.478657 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":496419712}
I20250831 07:59:28.567464 31801 raft_consensus.cc:1238] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:28.484863 31882 rpcz_store.cc:276] 0831 07:59:24.768811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:24.949423 (+180612us) service_pool.cc:225] Handling call
0831 07:59:24.984690 (+ 35267us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:26.465516 (+1480826us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.446530 (+1981014us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b540020d5d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:28.446705 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3153662208}
I20250831 07:59:28.399406 1669 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:28.490394 31807 rpcz_store.cc:276] 0831 07:59:26.302082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.385341 (+2083259us) service_pool.cc:225] Handling call
0831 07:59:28.385791 (+ 450us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.445929 (+ 60138us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.446165 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:28.449287 31883 rpcz_store.cc:269] 0831 07:59:25.580152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.235543 (+655391us) service_pool.cc:225] Handling call
0831 07:59:28.405712 (+2170169us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400410854. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:28.405881 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3049207680}
I20250831 07:59:28.472966 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:28.448586 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:28.450229 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 61998.8us
I20250831 07:59:28.572480 31729 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:28.502442 31726 raft_consensus.cc:1238] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 2. Current term is 3. Ops: []
W20250831 07:59:28.558135 32467 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47886 (request call id 433) took 4740 ms (client timeout 1665 ms). Trace:
I20250831 07:59:28.527110 31809 raft_consensus.cc:1238] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:28.563001 31802 rpcz_store.cc:276] 0831 07:59:24.264948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:24.265068 (+ 120us) service_pool.cc:225] Handling call
0831 07:59:24.265384 (+ 316us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:26.276809 (+2011425us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.562451 (+2285642us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b54002f0654. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:28.562595 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4451308288}
I20250831 07:59:28.452330 1910 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:28.499688 1877 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:28.539070 1918 log.cc:927] Time spent T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 1.469s user 0.003s sys 0.018s
I20250831 07:59:28.568898 31775 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:28.370455 1790 consensus_queue.cc:1046] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.102s
I20250831 07:59:28.489198 31808 rpcz_store.cc:276] 0831 07:59:26.567653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.392951 (+1825298us) service_pool.cc:225] Handling call
0831 07:59:28.393300 (+ 349us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.446822 (+ 53522us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.446980 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:28.538808 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 232) took 3391 ms. Trace:
W20250831 07:59:28.560123 1923 log.cc:927] Time spent T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 1.475s user 0.007s sys 0.017s
I20250831 07:59:28.595943 1891 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:28.598385 1894 consensus_queue.cc:1046] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20250831 07:59:28.565383 32464 rpcz_store.cc:276] 0831 07:59:26.641982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.642667 (+ 685us) service_pool.cc:225] Handling call
0831 07:59:28.210000 (+1567333us) spinlock_profiling.cc:243] Waited 6.45 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:28.210198 (+ 198us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:28.227059 (+ 16861us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:28.227100 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:28.227721 (+ 621us) log.cc:844] Serialized 26 byte log entry
0831 07:59:28.306350 (+ 78629us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:28.355104 (+ 48754us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.358873 (+ 3769us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:28.462409 (+103536us) raft_consensus.cc:1683] finished
0831 07:59:28.462428 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.462644 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15520,"raft.queue_time_us":7870,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":7354,"spinlock_wait_cycles":14841728,"thread_start_us":50020,"threads_started":1,"wal-append.queue_time_us":159854}
I20250831 07:59:28.597883 1275 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:28.560698 1843 raft_consensus.cc:738] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:28.601874 1264 consensus_queue.cc:1046] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250831 07:59:28.597687 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:28.603070 31726 raft_consensus.cc:1273] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20250831 07:59:28.604262 31698 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:28.595595 31807 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:28.706178 31726 pending_rounds.cc:85] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 3
I20250831 07:59:28.573757 31734 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:28.706676 31726 pending_rounds.cc:107] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250831 07:59:28.565562 31879 rpcz_store.cc:276] 0831 07:59:24.770225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:24.956969 (+186744us) service_pool.cc:225] Handling call
0831 07:59:24.961584 (+ 4615us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:26.225677 (+1264093us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.471377 (+2245700us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b5400299fd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:28.471549 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2901532928}
I20250831 07:59:28.560734 1946 raft_consensus.cc:738] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:28.648068 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 79278us
I20250831 07:59:28.539235 32469 raft_consensus.cc:1238] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:28.614989 31849 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:28.717272 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 102369us
W20250831 07:59:28.673434 1988 log.cc:927] Time spent T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
W20250831 07:59:28.674973 1917 log.cc:927] Time spent T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20250831 07:59:28.616639 31773 leader_election.cc:336] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.823s (SENT)
I20250831 07:59:28.573028 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250831 07:59:28.709672 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 105471us
W20250831 07:59:28.694830 31727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1327) took 2035 ms (client timeout 1559 ms). Trace:
I20250831 07:59:28.706974 31726 raft_consensus.cc:2887] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:28.712364 31729 raft_consensus.cc:2391] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 3: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:28.518747 1826 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:28.676185 1894 raft_consensus.cc:3053] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:28.714666 31775 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:28.824345 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 109763us
I20250831 07:59:28.799033 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:28.560263 1757 raft_consensus.cc:695] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 3 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:28.832013 31725 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:28.594835 1962 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:28.610649 1910 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:28.713515 31732 rpcz_store.cc:276] 0831 07:59:25.147644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:25.265828 (+118184us) service_pool.cc:225] Handling call
0831 07:59:25.266266 (+ 438us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:26.144490 (+878224us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:26.144554 (+ 64us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:26.195960 (+ 51406us) log.cc:844] Serialized 26 byte log entry
0831 07:59:26.565415 (+369455us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:27.669640 (+1104225us) log.cc:844] Serialized 14 byte log entry
0831 07:59:27.669896 (+ 256us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:27.669986 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:28.469537 (+799551us) spinlock_profiling.cc:243] Waited 43.6 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:28.534225 (+ 64688us) raft_consensus.cc:1683] finished
0831 07:59:28.538535 (+ 4310us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.538735 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":873949,"prepare.queue_time_us":513903,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":1550,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":76306,"spinlock_wait_cycles":100304256,"thread_start_us":1320630,"threads_started":2,"wal-append.queue_time_us":823853}
I20250831 07:59:28.846876 31732 raft_consensus.cc:1238] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: []
W20250831 07:59:28.819339 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 202726us
I20250831 07:59:28.562078 1950 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:28.573472 32467 rpcz_store.cc:269] 0831 07:59:23.817372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:23.817519 (+ 147us) service_pool.cc:225] Handling call
0831 07:59:24.811602 (+994083us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:28.557927 (+3746325us) spinlock_profiling.cc:243] Waited 2.56 s on lock 0x7b54006ffed4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:28.558061 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5899330688}
I20250831 07:59:28.687259 1967 raft_consensus.cc:3053] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Stepping down as leader of term 6
W20250831 07:59:28.821327 31727 rpcz_store.cc:269] 0831 07:59:26.659657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.057311 (+397654us) service_pool.cc:225] Handling call
0831 07:59:28.694574 (+1637263us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540027f954. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:28.694737 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2534066560}
I20250831 07:59:28.822165 31726 pending_rounds.cc:107] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20250831 07:59:28.638978 1939 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:28.825191 31775 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:28.683503 31800 raft_consensus.cc:1238] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
W20250831 07:59:28.826179 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:28.826476 31808 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:28.629323 1240 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:28.858990 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:28.673637 1600 raft_consensus.cc:3053] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Stepping down as leader of term 4
W20250831 07:59:28.837409 31728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1303) took 4088 ms (client timeout 1938 ms). Trace:
I20250831 07:59:28.863744 31875 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
W20250831 07:59:28.863866 31728 rpcz_store.cc:269] 0831 07:59:24.749446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:24.750503 (+ 1057us) service_pool.cc:225] Handling call
0831 07:59:28.837131 (+4086628us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54001dfe54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:28.837316 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3259424512}
I20250831 07:59:28.571305 1940 consensus_queue.cc:1046] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.241s
W20250831 07:59:28.855835 31627 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59666 (request call id 236) took 15990 ms (client timeout 15000 ms). Trace:
I20250831 07:59:28.844063 31879 raft_consensus.cc:1273] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250831 07:59:28.718355 31849 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:28.673849 1669 raft_consensus.cc:2747] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:28.823721 1894 raft_consensus.cc:738] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:28.848663 31729 raft_consensus.cc:1238] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:28.883757 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1822) took 1124 ms. Trace:
I20250831 07:59:28.884228 31729 rpcz_store.cc:276] 0831 07:59:27.759513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.821255 (+1061742us) service_pool.cc:225] Handling call
0831 07:59:28.848163 (+ 26908us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.883255 (+ 35092us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.883683 (+ 428us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:28.539947 31716 tablet_service.cc:1515] Processing DeleteTablet for tablet 26525016efb247c3b1c2bf7fec0e1658 with delete_type TABLET_DATA_TOMBSTONED (TS a0d845d01c8c4894895f66e3beae298c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57666
I20250831 07:59:28.675237 1879 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:28.901647 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:28.922528 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:28.686215 1884 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:28.869606 1855 consensus_queue.cc:1046] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.082s
I20250831 07:59:28.690217 1498 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:28.717489 1813 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:28.861639 32435 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:28.856559 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:28.612210 1871 consensus_queue.cc:1046] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.228s
I20250831 07:59:28.798460 1790 raft_consensus.cc:2747] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:28.674388 1791 consensus_queue.cc:1046] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.175s
I20250831 07:59:28.870251 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1825) took 1109 ms. Trace:
W20250831 07:59:28.875150 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 156872us
I20250831 07:59:28.839835 1962 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:28.847785 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1823) took 1088 ms. Trace:
I20250831 07:59:28.688225 1378 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:28.612946 1860 consensus_queue.cc:1035] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.236s
I20250831 07:59:28.881727 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:28.836275 1880 raft_consensus.cc:2747] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:28.713456 1881 consensus_queue.cc:1046] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.149s
I20250831 07:59:28.857040 1867 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:28.928755 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1796) took 4499 ms. Trace:
I20250831 07:59:28.851604 32463 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 376) took 3851 ms. Trace:
I20250831 07:59:28.854722 31726 raft_consensus.cc:2887] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:28.869308 1600 raft_consensus.cc:738] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:28.718459 1802 consensus_queue.cc:1046] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.163s
I20250831 07:59:28.713842 1275 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:29.147922 31734 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
W20250831 07:59:29.147672 31730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1804) took 1405 ms (client timeout 1590 ms). Trace:
W20250831 07:59:29.025918 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 200799us
I20250831 07:59:29.004406 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 231) took 3856 ms. Trace:
I20250831 07:59:29.064725 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 320) took 1117 ms. Trace:
I20250831 07:59:28.886965 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:28.888126 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:28.446527 31806 rpcz_store.cc:276] 0831 07:59:24.787469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.213229 (+3425760us) service_pool.cc:225] Handling call
0831 07:59:28.214286 (+ 1057us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.214477 (+ 191us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 07:59:28.214741 (+ 264us) log.cc:844] Serialized 14 byte log entry
0831 07:59:28.407260 (+192519us) log.cc:844] Serialized 14 byte log entry
0831 07:59:28.407438 (+ 178us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:28.407494 (+ 56us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:28.407542 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.407608 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.407706 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":259,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":71065,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"spinlock_wait_cycles":11448448,"thread_start_us":191823,"threads_started":1}
I20250831 07:59:29.013922 31729 raft_consensus.cc:1238] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:29.540303 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1828) took 1768 ms. Trace:
I20250831 07:59:28.960685 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.540809 31729 rpcz_store.cc:276] 0831 07:59:27.835871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.885762 (+1049891us) service_pool.cc:225] Handling call
0831 07:59:29.013438 (+127676us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:29.539972 (+526534us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.540237 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:29.538774 31803 rpcz_store.cc:276] 0831 07:59:24.429443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.037530 (+3608087us) service_pool.cc:225] Handling call
0831 07:59:28.037843 (+ 313us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.195181 (+157338us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.906847 (+711666us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b5400028d54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:28.928670 (+ 21823us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":360576768}
I20250831 07:59:29.543365 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.539029 31804 rpcz_store.cc:276] 0831 07:59:27.947701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.488745 (+541044us) service_pool.cc:225] Handling call
0831 07:59:28.489852 (+ 1107us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.489992 (+ 140us) raft_consensus.cc:1532] Early marking committed up to index 6
0831 07:59:28.713531 (+223539us) log.cc:844] Serialized 14 byte log entry
0831 07:59:29.064235 (+350704us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:29.064307 (+ 72us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:29.064392 (+ 85us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.064496 (+ 104us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.064654 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":450,"prepare.queue_time_us":110933,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":16358,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":8679,"thread_start_us":568716,"threads_started":2,"wal-append.queue_time_us":503509}
I20250831 07:59:29.543479 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.544411 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:28.964074 31725 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
W20250831 07:59:29.094182 1918 log.cc:927] Time spent T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20250831 07:59:29.545500 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.540181 31775 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:29.469112 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:29.538542 31733 rpcz_store.cc:276] 0831 07:59:25.147460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:25.263865 (+116405us) service_pool.cc:225] Handling call
0831 07:59:25.466797 (+202932us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 07:59:26.233411 (+766614us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:26.233443 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 07:59:26.233984 (+ 541us) log.cc:844] Serialized 52 byte log entry
0831 07:59:26.665319 (+431335us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:27.311100 (+645781us) log.cc:844] Serialized 14 byte log entry
0831 07:59:27.345786 (+ 34686us) log.cc:844] Serialized 14 byte log entry
0831 07:59:27.346060 (+ 274us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:27.346180 (+ 120us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:28.204607 (+858427us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b18003d0ea0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b8bf 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:59:28.538484 (+333877us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:29.004088 (+465604us) raft_consensus.cc:1683] finished
0831 07:59:29.004122 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.004330 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":866436,"prepare.queue_time_us":415581,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":73,"raft.queue_time_us":800320,"raft.run_cpu_time_us":325,"raft.run_wall_time_us":60217,"spinlock_wait_cycles":18340480,"thread_start_us":916619,"threads_started":2,"wal-append.queue_time_us":610546}
I20250831 07:59:29.550233 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.540983 31800 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:29.553715 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.554752 31800 raft_consensus.cc:2408] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250831 07:59:29.554045 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.537915 31734 pending_rounds.cc:85] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 6
I20250831 07:59:29.556988 31734 pending_rounds.cc:107] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20250831 07:59:29.557330 31734 raft_consensus.cc:2887] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:29.537925 31732 rpcz_store.cc:276] 0831 07:59:27.759857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.845889 (+1086032us) service_pool.cc:225] Handling call
0831 07:59:28.846623 (+ 734us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.847461 (+ 838us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.847719 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:29.558743 31808 raft_consensus.cc:1273] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 07:59:29.560475 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 238) took 1649 ms. Trace:
W20250831 07:59:29.539387 31730 rpcz_store.cc:269] 0831 07:59:27.745389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.775007 (+ 29618us) service_pool.cc:225] Handling call
0831 07:59:29.147343 (+1372336us) spinlock_profiling.cc:243] Waited 58.1 ms on lock 0x7b54003e3854. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:29.147590 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":133646208}
I20250831 07:59:29.567852 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 239) took 1463 ms. Trace:
I20250831 07:59:29.568342 31730 rpcz_store.cc:276] 0831 07:59:28.104821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:29.566819 (+1461998us) service_pool.cc:225] Handling call
0831 07:59:29.567154 (+ 335us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:29.567396 (+ 242us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:29.567442 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:29.567503 (+ 61us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:29.567564 (+ 61us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.567644 (+ 80us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.567769 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:29.565841 31732 rpcz_store.cc:276] 0831 07:59:27.911526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:29.558508 (+1646982us) service_pool.cc:225] Handling call
0831 07:59:29.558831 (+ 323us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:29.559962 (+ 1131us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:29.560092 (+ 130us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:29.560152 (+ 60us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:29.560198 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.560271 (+ 73us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.560410 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:29.573944 31800 raft_consensus.cc:1238] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
W20250831 07:59:29.576691 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 166190us
I20250831 07:59:29.544893 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.580412 2007 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:29.581112 2007 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:29.587554 1991 log.cc:927] Time spent T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.082s user 0.004s sys 0.001s
I20250831 07:59:29.590353 1843 consensus_queue.cc:260] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:29.591877 1843 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:29.593598 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43230 (request call id 248) took 1194 ms. Trace:
I20250831 07:59:29.595790 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1821) took 1840 ms. Trace:
I20250831 07:59:29.670511 31725 rpcz_store.cc:276] 0831 07:59:28.399056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.487890 (+ 88834us) service_pool.cc:225] Handling call
0831 07:59:29.593354 (+1105464us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b5400310e54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:29.593519 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":765619072}
I20250831 07:59:29.670663 31734 rpcz_store.cc:276] 0831 07:59:27.755670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.709253 (+953583us) service_pool.cc:225] Handling call
0831 07:59:29.147516 (+438263us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:29.557896 (+410380us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.595309 (+ 37413us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b54003e3850. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:29.595502 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1007141248}
I20250831 07:59:29.672979 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 245) took 1566 ms. Trace:
I20250831 07:59:29.673120 31734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 247) took 1540 ms. Trace:
W20250831 07:59:29.597628 31698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.020s sys 0.046s
I20250831 07:59:29.599502 2007 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:29.603646 1939 consensus_meta.cc:287] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.734s user 0.012s sys 0.000s
W20250831 07:59:29.608274 32464 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.006s sys 0.014s
W20250831 07:59:29.609360 32433 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.009s sys 0.012s
W20250831 07:59:29.609861 31848 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.007s sys 0.013s
W20250831 07:59:29.097965 1984 log.cc:927] Time spent T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
W20250831 07:59:29.697186 31848 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.634s user 0.008s sys 0.014s
W20250831 07:59:29.698244 31848 connection.cc:232] RPC call timeout handler was delayed by -2.0192390480000313s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1715s for a response
I20250831 07:59:28.903761 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:29.610581 1804 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.009s sys 0.028s
I20250831 07:59:29.613692 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 242) took 1508 ms. Trace:
I20250831 07:59:29.290180 1385 raft_consensus.cc:3053] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Stepping down as leader of term 6
W20250831 07:59:29.615468 1882 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.011s sys 0.010s
I20250831 07:59:29.615051 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1818) took 1860 ms. Trace:
I20250831 07:59:29.615391 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 237) took 1704 ms. Trace:
W20250831 07:59:29.702255 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 840634us
I20250831 07:59:29.063012 1949 consensus_queue.cc:1046] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.630s
I20250831 07:59:29.705526 31733 rpcz_store.cc:276] 0831 07:59:27.911156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:29.549136 (+1637980us) service_pool.cc:225] Handling call
0831 07:59:29.549585 (+ 449us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:29.549999 (+ 414us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:29.550038 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:29.550641 (+ 603us) log.cc:844] Serialized 26 byte log entry
0831 07:59:29.585552 (+ 34911us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:29.614218 (+ 28666us) log.cc:844] Serialized 14 byte log entry
0831 07:59:29.615089 (+ 871us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.615159 (+ 70us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:29.615208 (+ 49us) raft_consensus.cc:1683] finished
0831 07:59:29.615220 (+ 12us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.615334 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":26686,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":49385,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":210,"thread_start_us":62759,"threads_started":2,"wal-append.queue_time_us":32679}
I20250831 07:59:29.617203 31849 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:29.666469 1935 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
W20250831 07:59:29.707654 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 90565.3us
I20250831 07:59:29.618209 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1827) took 1856 ms. Trace:
W20250831 07:59:29.319835 1989 log.cc:927] Time spent T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.201s user 0.001s sys 0.000s
I20250831 07:59:29.706789 1871 raft_consensus.cc:2802] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:29.377313 1950 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:29.216298 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 380) took 2421 ms. Trace:
I20250831 07:59:29.651077 1867 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:29.712231 31726 rpcz_store.cc:276] 0831 07:59:27.754799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.574314 (+819515us) service_pool.cc:225] Handling call
0831 07:59:28.602767 (+ 28453us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:29.614719 (+1011952us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.614978 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:29.652503 1940 consensus_queue.cc:1046] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250831 07:59:29.652828 1891 raft_consensus.cc:3053] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:29.650672 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:29.653781 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 113726us
I20250831 07:59:29.655478 31727 rpcz_store.cc:276] 0831 07:59:27.760742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.854907 (+1094165us) service_pool.cc:225] Handling call
0831 07:59:28.867798 (+ 12891us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.869902 (+ 2104us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:28.869932 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:28.869979 (+ 47us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:28.870026 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.870084 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.870186 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1684,"raft.queue_time_us":2514,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":8988}
W20250831 07:59:29.657490 31699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.693s user 0.014s sys 0.049s
I20250831 07:59:29.322981 32467 raft_consensus.cc:1238] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 7. Current term is 8. Ops: []
I20250831 07:59:28.886281 1240 raft_consensus.cc:2802] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 07:59:29.674611 31698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.687s user 0.020s sys 0.046s
I20250831 07:59:29.673462 31725 rpcz_store.cc:276] 0831 07:59:28.106623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:29.671873 (+1565250us) service_pool.cc:225] Handling call
0831 07:59:29.672283 (+ 410us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:29.672508 (+ 225us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:29.672548 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:29.672607 (+ 59us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:29.672659 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.672734 (+ 75us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.672910 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:29.673616 31734 rpcz_store.cc:276] 0831 07:59:28.132975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:29.672113 (+1539138us) service_pool.cc:225] Handling call
0831 07:59:29.672478 (+ 365us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:29.672706 (+ 228us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:29.672746 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:29.672804 (+ 58us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:29.672855 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.672924 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.673055 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:29.724705 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:29.080083 1927 log.cc:927] Time spent T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 2.001s user 0.022s sys 0.003s
I20250831 07:59:28.903505 1585 raft_consensus.cc:2747] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:28.855589 1967 raft_consensus.cc:738] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:29.692085 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1750) took 1118 ms. Trace:
I20250831 07:59:29.289685 1396 raft_consensus.cc:3053] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Stepping down as leader of term 5
W20250831 07:59:29.699071 31848 connection.cc:232] RPC call timeout handler was delayed by -1.881539048000036s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18680000000000002s for a response
W20250831 07:59:29.610139 1966 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.000s sys 0.021s
I20250831 07:59:29.110436 1826 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:29.148262 1853 raft_consensus.cc:3053] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:29.651371 1378 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:28.961184 1740 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:29.703416 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:59:29.619843 1737 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:29.063629 1894 consensus_queue.cc:260] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.619946 1962 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:28.987748 1735 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:29.745324 1735 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.659035 1947 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:29.708428 31807 raft_consensus.cc:1238] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:29.721220 1896 consensus_queue.cc:1046] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
W20250831 07:59:29.691418 32464 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.638s user 0.007s sys 0.015s
I20250831 07:59:29.708835 31728 rpcz_store.cc:276] 0831 07:59:27.761308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.864836 (+1103528us) service_pool.cc:225] Handling call
0831 07:59:28.865166 (+ 330us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:28.865516 (+ 350us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:28.865549 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:28.866128 (+ 579us) log.cc:844] Serialized 26 byte log entry
0831 07:59:29.290425 (+424297us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:29.290473 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.290570 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:29.617938 (+327368us) raft_consensus.cc:1683] finished
0831 07:59:29.617967 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.618135 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24134,"raft.queue_time_us":4290,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":4569,"thread_start_us":331486,"threads_started":1,"wal-append.queue_time_us":729706}
I20250831 07:59:29.710793 31849 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:29.704625 31730 rpcz_store.cc:276] 0831 07:59:28.105751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:29.569589 (+1463838us) service_pool.cc:225] Handling call
0831 07:59:29.569959 (+ 370us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:29.578529 (+ 8570us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:29.578574 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:29.579152 (+ 578us) log.cc:844] Serialized 26 byte log entry
0831 07:59:29.589201 (+ 10049us) raft_consensus.cc:1643] Marking committed up to 8
0831 07:59:29.612301 (+ 23100us) log.cc:844] Serialized 14 byte log entry
0831 07:59:29.613259 (+ 958us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.613377 (+ 118us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:29.613448 (+ 71us) raft_consensus.cc:1683] finished
0831 07:59:29.613467 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.613624 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":23275,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":28762,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":156,"thread_start_us":40258,"threads_started":3,"wal-append.queue_time_us":11678}
W20250831 07:59:29.696928 32433 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s user 0.009s sys 0.012s
I20250831 07:59:29.652101 1881 raft_consensus.cc:3053] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Stepping down as leader of term 8
I20250831 07:59:29.712662 31802 raft_consensus.cc:1238] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:29.732683 1396 raft_consensus.cc:738] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:29.704423 1804 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s user 0.009s sys 0.029s
W20250831 07:59:29.720306 31699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.756s user 0.014s sys 0.050s
W20250831 07:59:29.706993 1882 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.654s user 0.012s sys 0.010s
I20250831 07:59:29.722182 31806 raft_consensus.cc:2391] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 3: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:29.030014 1877 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:29.511693 1993 log.cc:927] Time spent T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.211s user 0.000s sys 0.000s
W20250831 07:59:29.725194 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:29.726460 31726 raft_consensus.cc:1273] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250831 07:59:29.079195 1965 consensus_queue.cc:1046] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.635s
I20250831 07:59:29.707201 1385 raft_consensus.cc:738] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:29.730325 31727 raft_consensus.cc:1273] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:29.709455 1871 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:29.733043 31848 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:29.781860 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 48907us
I20250831 07:59:29.782413 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 236) took 4633 ms. Trace:
I20250831 07:59:29.782791 31731 rpcz_store.cc:276] 0831 07:59:25.148692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.431392 (+1282700us) service_pool.cc:225] Handling call
0831 07:59:26.459713 (+ 28321us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:26.460097 (+ 384us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:26.460136 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:26.460737 (+ 601us) log.cc:844] Serialized 26 byte log entry
0831 07:59:26.798971 (+338234us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:26.799019 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:26.799910 (+ 891us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:28.799480 (+1999570us) spinlock_profiling.cc:243] Waited 45.4 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:29.780656 (+981176us) raft_consensus.cc:1683] finished
0831 07:59:29.780681 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.782352 (+ 1671us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":97,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":6361,"spinlock_wait_cycles":104380160,"thread_start_us":330374,"threads_started":1,"wal-append.queue_time_us":613516}
I20250831 07:59:29.784003 1882 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:29.784736 1871 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:29.785499 1385 consensus_queue.cc:260] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.786872 1385 raft_consensus.cc:3058] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:29.787262 31628 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43950 (request call id 122) took 16987 ms (client timeout 15000 ms). Trace:
I20250831 07:59:29.733654 31775 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:29.080423 1757 consensus_queue.cc:237] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.788717 1880 raft_consensus.cc:2747] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250831 07:59:29.788867 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 55301.2us
I20250831 07:59:29.789743 1877 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:29.790441 31801 raft_consensus.cc:1238] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 5. Current term is 6. Ops: []
W20250831 07:59:29.790627 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:29.792534 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 324) took 1384 ms. Trace:
W20250831 07:59:29.787675 31628 rpcz_store.cc:269] 0831 07:59:12.800066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:12.800267 (+ 201us) service_pool.cc:225] Handling call
0831 07:59:29.787169 (+16986902us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:59:29.114721 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:59:29.115545 (+ 824us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:59:29.115609 (+ 64us) write_op.cc:435] Acquired schema lock
0831 07:59:29.115649 (+ 40us) tablet.cc:660] Decoding operations
0831 07:59:29.293211 (+177562us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:59:29.308384 (+ 15173us) write_op.cc:641] Partition lock acquired for write op
0831 07:59:29.309885 (+ 1501us) tablet.cc:683] Acquiring locks for 3 operations
0831 07:59:29.534849 (+224964us) tablet.cc:699] Row locks acquired
0831 07:59:29.534900 (+ 51us) write_op.cc:260] PREPARE: finished
0831 07:59:29.535159 (+ 259us) write_op.cc:270] Start()
0831 07:59:29.535376 (+ 217us) write_op.cc:276] Timestamp: P: 1756627169535066 usec, L: 0
0831 07:59:29.535407 (+ 31us) op_driver.cc:348] REPLICATION: starting
0831 07:59:29.545080 (+ 9673us) log.cc:844] Serialized 1293 byte log entry
0831 07:59:29.614480 (+ 69400us) op_driver.cc:464] REPLICATION: finished
0831 07:59:29.618507 (+ 4027us) write_op.cc:301] APPLY: starting
0831 07:59:29.618639 (+ 132us) tablet.cc:1365] starting BulkCheckPresence
0831 07:59:29.618836 (+ 197us) tablet.cc:1368] finished BulkCheckPresence
0831 07:59:29.619042 (+ 206us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:59:29.731510 (+112468us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:59:29.761220 (+ 29710us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0831 07:59:29.765462 (+ 4242us) write_op.cc:312] APPLY: finished
0831 07:59:29.769281 (+ 3819us) log.cc:844] Serialized 35 byte log entry
0831 07:59:29.770238 (+ 957us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:29.770334 (+ 96us) write_op.cc:454] Released schema lock
0831 07:59:29.770390 (+ 56us) write_op.cc:341] FINISH: Updating metrics
0831 07:59:29.782083 (+ 11693us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:29.782118 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7129081,"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":3701,"apply.run_cpu_time_us":3610,"apply.run_wall_time_us":162865,"num_ops":3,"prepare.queue_time_us":584507,"prepare.run_cpu_time_us":7493,"prepare.run_wall_time_us":498217,"raft.queue_time_us":39330,"raft.run_cpu_time_us":2011,"raft.run_wall_time_us":4048,"replication_time_us":78883,"thread_start_us":340070,"threads_started":4,"wal-append.queue_time_us":27538}]]}
I20250831 07:59:29.792884 31801 rpcz_store.cc:276] 0831 07:59:28.408140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.568338 (+160198us) service_pool.cc:225] Handling call
0831 07:59:28.621199 (+ 52861us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:29.790863 (+1169664us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.792331 (+ 1468us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400084dd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:29.792481 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2688617728}
I20250831 07:59:29.053259 1946 consensus_queue.cc:260] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.731740 31883 rpcz_store.cc:276] 0831 07:59:28.573615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.573810 (+ 195us) service_pool.cc:225] Handling call
0831 07:59:28.574229 (+ 419us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:28.612961 (+ 38732us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:28.612998 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:28.640492 (+ 27494us) log.cc:844] Serialized 26 byte log entry
0831 07:59:28.990978 (+350486us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:28.991033 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.991127 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:29.586935 (+595808us) spinlock_profiling.cc:243] Waited 30.8 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:29.660043 (+ 73108us) raft_consensus.cc:1683] finished
0831 07:59:29.660091 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.691992 (+ 31901us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29136,"raft.queue_time_us":29329,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":19915,"spinlock_wait_cycles":70886784,"thread_start_us":316107,"threads_started":1,"wal-append.queue_time_us":728208}
I20250831 07:59:29.783596 31848 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:29.721778 1897 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.024s sys 0.089s
W20250831 07:59:29.795569 1897 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.751s user 0.024s sys 0.089s
I20250831 07:59:29.796133 1897 raft_consensus.cc:2747] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:29.713376 1997 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:29.796825 1997 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:29.722348 1935 raft_consensus.cc:695] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:29.797200 1997 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:29.797767 1935 consensus_queue.cc:237] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:29.798782 1894 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:29.028510 1401 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:29.800069 1401 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.800448 1946 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:29.802127 1401 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:29.805079 1997 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.765370 1396 consensus_queue.cc:260] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.806262 1396 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:29.807251 1997 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:29.808010 1940 consensus_queue.cc:1162] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a0d845d01c8c4894895f66e3beae298c" responder_term: 5 status { last_received { term: 5 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 5 index: 6 } }
W20250831 07:59:29.734198 1966 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.641s user 0.000s sys 0.021s
W20250831 07:59:28.871603 31627 rpcz_store.cc:269] 0831 07:59:12.560549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:12.560711 (+ 162us) service_pool.cc:225] Handling call
0831 07:59:28.550736 (+15990025us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:59:13.779596 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:59:13.780200 (+ 604us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:59:13.780236 (+ 36us) write_op.cc:435] Acquired schema lock
0831 07:59:13.780251 (+ 15us) tablet.cc:660] Decoding operations
0831 07:59:13.807905 (+ 27654us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:59:13.807994 (+ 89us) write_op.cc:641] Partition lock acquired for write op
0831 07:59:13.808041 (+ 47us) tablet.cc:683] Acquiring locks for 16 operations
0831 07:59:13.809716 (+ 1675us) tablet.cc:699] Row locks acquired
0831 07:59:13.809743 (+ 27us) write_op.cc:260] PREPARE: finished
0831 07:59:13.809939 (+ 196us) write_op.cc:270] Start()
0831 07:59:13.810087 (+ 148us) write_op.cc:276] Timestamp: P: 1756627153809871 usec, L: 0
0831 07:59:13.810119 (+ 32us) op_driver.cc:348] REPLICATION: starting
0831 07:59:13.811043 (+ 924us) log.cc:844] Serialized 6284 byte log entry
0831 07:59:14.602524 (+791481us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:15.080739 (+478215us) op_driver.cc:464] REPLICATION: finished
0831 07:59:15.353165 (+272426us) write_op.cc:301] APPLY: starting
0831 07:59:15.355809 (+ 2644us) tablet.cc:1365] starting BulkCheckPresence
0831 07:59:15.416053 (+ 60244us) tablet.cc:1368] finished BulkCheckPresence
0831 07:59:15.416086 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:59:15.676529 (+260443us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:59:15.778903 (+102374us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0831 07:59:15.778950 (+ 47us) write_op.cc:312] APPLY: finished
0831 07:59:15.779314 (+ 364us) log.cc:844] Serialized 113 byte log entry
0831 07:59:15.824214 (+ 44900us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:15.873456 (+ 49242us) write_op.cc:454] Released schema lock
0831 07:59:15.884341 (+ 10885us) write_op.cc:341] FINISH: Updating metrics
0831 07:59:15.911643 (+ 27302us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:15.914480 (+ 2837us) write_op.cc:454] Released schema lock
0831 07:59:16.404324 (+489844us) spinlock_profiling.cc:243] Waited 1.06 ms on lock 0x7b54000af354. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffa2ea 00007fdce7f97644 00007fdce7f975f3 00007fdce7f97568 00007fdce7f97523 00007fdce7f9680c 00007fdce7fc04a6 00007fdce7fa846b 00007fdce7f92acd 00007fdce7f964a5
Metrics: {"mutex_wait_us":250473,"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":255687,"apply.run_cpu_time_us":12252,"apply.run_wall_time_us":558549,"num_ops":16,"prepare.queue_time_us":230219,"prepare.run_cpu_time_us":22344,"prepare.run_wall_time_us":103743,"raft.queue_time_us":333664,"raft.run_cpu_time_us":43708,"raft.run_wall_time_us":1409687,"replication_time_us":1239318,"spinlock_wait_cycles":52868736,"thread_start_us":470091,"threads_started":4,"wal-append.queue_time_us":376419,"wal-append.run_cpu_time_us":53169,"wal-append.run_wall_time_us":2707185}]]}
I20250831 07:59:29.619635 32463 rpcz_store.cc:276] 0831 07:59:24.991686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:25.021683 (+ 29997us) service_pool.cc:225] Handling call
0831 07:59:25.102340 (+ 80657us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:25.194586 (+ 92246us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:25.199508 (+ 4922us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:25.202527 (+ 3019us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:25.210683 (+ 8156us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:28.817414 (+3606731us) spinlock_profiling.cc:243] Waited 77.5 ms on lock 0x7b540068b354. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:28.823051 (+ 5637us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:28.842525 (+ 19474us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":178202496}
I20250831 07:59:28.882432 1901 consensus_queue.cc:1035] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.769s
W20250831 07:59:29.713542 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:29.749951 1881 raft_consensus.cc:738] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:29.740056 1826 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.752305 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1793) took 5392 ms. Trace:
I20250831 07:59:29.752833 1896 raft_consensus.cc:3053] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:29.754843 1867 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:29.755581 2000 raft_consensus.cc:3053] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:29.756151 1275 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:29.735749 1853 raft_consensus.cc:738] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:29.758440 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 47751us
I20250831 07:59:29.713131 32469 rpcz_store.cc:276] 0831 07:59:26.794596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:26.797034 (+ 2438us) service_pool.cc:225] Handling call
0831 07:59:28.494432 (+1697398us) spinlock_profiling.cc:243] Waited 9.79 ms on lock 0x7b5800068198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:28.494647 (+ 215us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.799499 (+304852us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.216070 (+416571us) spinlock_profiling.cc:243] Waited 31.7 ms on lock 0x7b540031a1d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:29.216235 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":95522944}
I20250831 07:59:29.713562 1600 consensus_queue.cc:260] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:29.759498 1624 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:29.733685 1860 consensus_queue.cc:1046] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20250831 07:59:29.764622 1967 consensus_queue.cc:260] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.827606 1896 raft_consensus.cc:738] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:29.761622 1947 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:29.768164 32436 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:29.854768 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 86676.5us
I20250831 07:59:29.768692 1735 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:29.776350 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 251) took 1099 ms. Trace:
I20250831 07:59:29.862272 31726 rpcz_store.cc:276] 0831 07:59:28.676793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:29.713151 (+1036358us) service_pool.cc:225] Handling call
0831 07:59:29.726231 (+ 13080us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:29.776008 (+ 49777us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.776279 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:29.863268 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:29.796002 31848 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:29.864269 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.864830 31726 raft_consensus.cc:2466] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:29.839781 1624 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:29.719580 1891 raft_consensus.cc:738] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:29.808163 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:29.879773 1910 consensus_meta.cc:287] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 1.017s user 0.010s sys 0.001s
I20250831 07:59:29.810523 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.812661 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.814229 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1829) took 1977 ms. Trace:
I20250831 07:59:29.742388 31803 raft_consensus.cc:3053] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:29.822791 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.823683 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.894950 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.895584 31725 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:29.896210 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.897027 31806 raft_consensus.cc:2408] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 07:59:29.824330 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:29.895160 31803 raft_consensus.cc:738] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:29.897465 31802 raft_consensus.cc:2374] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5. Re-sending same reply.
I20250831 07:59:29.898319 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.898865 31802 raft_consensus.cc:2391] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:29.825533 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.825039 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.822525 1949 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:29.831117 31698 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:29.905623 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 74593.1us
I20250831 07:59:29.905845 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.824918 1401 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:29.816210 1804 raft_consensus.cc:3053] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:29.834437 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.843451 1998 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20250831 07:59:29.834662 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:29.905155 31803 consensus_queue.cc:260] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.835963 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.913406 31803 raft_consensus.cc:3058] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:29.825838 31805 rpcz_store.cc:276] 0831 07:59:24.359792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:27.687983 (+3328191us) service_pool.cc:225] Handling call
0831 07:59:27.688353 (+ 370us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 07:59:27.699835 (+ 11482us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:27.699872 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 07:59:27.700500 (+ 628us) log.cc:844] Serialized 52 byte log entry
0831 07:59:28.375137 (+674637us) spinlock_profiling.cc:243] Waited 6.11 ms on lock 0x7b180036f160. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce507e0f7 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdce7c3b2bb 00007fdce7c40dc3 00007fdce7c41171 00007fdce7fd009f
0831 07:59:28.510319 (+135182us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:29.001877 (+491558us) log.cc:844] Serialized 14 byte log entry
0831 07:59:29.011888 (+ 10011us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.011964 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:29.749202 (+737238us) raft_consensus.cc:1683] finished
0831 07:59:29.752094 (+ 2892us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.752249 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11699,"prepare.queue_time_us":90702,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":14205,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":34180,"spinlock_wait_cycles":14060288,"thread_start_us":535895,"threads_started":2,"wal-append.queue_time_us":1121899}
I20250831 07:59:29.914574 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.915326 31805 raft_consensus.cc:2408] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 07:59:29.866521 1940 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:29.837747 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.825026 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:29.845369 1378 consensus_queue.cc:1035] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.052s
I20250831 07:59:29.916923 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.918223 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.919070 31807 raft_consensus.cc:2391] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:29.733507 1840 consensus_queue.cc:1035] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.017s
I20250831 07:59:29.935541 1871 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.936918 1940 raft_consensus.cc:695] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:29.833887 2000 raft_consensus.cc:738] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:29.937872 1940 consensus_queue.cc:237] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.938383 2000 consensus_queue.cc:260] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.822454 1737 consensus_queue.cc:1035] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20250831 07:59:29.865645 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:29.940135 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.940861 31808 raft_consensus.cc:2366] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6.
I20250831 07:59:29.941175 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:29.941700 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.942399 31808 raft_consensus.cc:2391] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:29.943130 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.940168 2000 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:29.911660 1884 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20250831 07:59:29.836851 1791 consensus_queue.cc:1046] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20250831 07:59:29.833946 1606 consensus_queue.cc:1046] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.020s
I20250831 07:59:29.870203 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 249) took 1369 ms. Trace:
I20250831 07:59:29.869882 1962 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.972941 31733 rpcz_store.cc:276] 0831 07:59:28.500447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:29.707088 (+1206641us) service_pool.cc:225] Handling call
0831 07:59:29.724668 (+ 17580us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:29.838645 (+113977us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:29.838674 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:29.838714 (+ 40us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:29.838747 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.869988 (+ 31241us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b5400350cd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:29.870028 (+ 40us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.870149 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":261686144}
I20250831 07:59:29.974073 1606 raft_consensus.cc:3053] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:29.974325 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:29.875118 1600 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:29.974709 1962 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:29.851377 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:29.875800 1853 consensus_queue.cc:260] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.975777 32462 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:29.976244 1853 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:29.976524 1855 consensus_queue.cc:1162] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a0d845d01c8c4894895f66e3beae298c" responder_term: 4 status { last_received { term: 4 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 4 index: 6 } }
I20250831 07:59:29.976893 31809 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:29.978330 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 328) took 1277 ms. Trace:
I20250831 07:59:29.978737 31809 rpcz_store.cc:276] 0831 07:59:28.701263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.711351 (+ 10088us) service_pool.cc:225] Handling call
0831 07:59:28.711696 (+ 345us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:29.977401 (+1265705us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.978060 (+ 659us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54003e12d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:29.978264 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2909231872}
I20250831 07:59:29.880568 1891 consensus_queue.cc:260] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.880843 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.883059 1826 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:29.853711 1881 consensus_queue.cc:260] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.980305 31731 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:29.980649 1891 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:29.980825 1881 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:29.981156 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:29.981737 31731 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:29.881489 1624 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:29.894230 31729 rpcz_store.cc:276] 0831 07:59:27.836608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:29.650907 (+1814299us) service_pool.cc:225] Handling call
0831 07:59:29.651358 (+ 451us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:29.651813 (+ 455us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:29.651857 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:29.652418 (+ 561us) log.cc:844] Serialized 26 byte log entry
0831 07:59:29.731182 (+ 78764us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:29.731209 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:29.731253 (+ 44us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:29.814026 (+ 82773us) raft_consensus.cc:1683] finished
0831 07:59:29.814045 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:29.814166 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":108,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":100,"thread_start_us":78329,"threads_started":1,"wal-append.queue_time_us":104618}
I20250831 07:59:29.899592 31727 raft_consensus.cc:2408] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 07:59:29.841099 1882 consensus_queue.cc:1046] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20250831 07:59:29.809695 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.984158 31730 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:29.984457 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.984953 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.985651 31730 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:29.986485 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:29.986944 31730 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 6
I20250831 07:59:29.987375 31804 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:29.988692 31698 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:29.985044 31727 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:29.989740 1826 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:29.990367 1826 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:29.990741 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.895696 31734 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:29.991781 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.992551 31734 raft_consensus.cc:2408] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 10 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 07:59:29.993253 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.993779 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.994164 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.983736 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.995117 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.909264 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:29.995724 31729 raft_consensus.cc:2391] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:29.996479 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.841015 1897 consensus_queue.cc:1046] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20250831 07:59:29.911892 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:29.998930 1962 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:29.999208 31725 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:29.999648 1962 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:29.916023 31800 raft_consensus.cc:2391] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:30.001210 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.002197 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.002944 1962 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:30.003142 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.765566 31732 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:29.906919 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.905217 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:30.004623 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.005295 31802 raft_consensus.cc:2391] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:29.912463 1804 raft_consensus.cc:738] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:29.855754 1868 consensus_queue.cc:1162] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7512ba2ed3a94982a22a75f51a647e65" responder_term: 7 status { last_received { term: 6 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250831 07:59:29.925683 1947 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:30.005988 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.006626 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.006913 31809 raft_consensus.cc:2391] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:30.007357 31802 raft_consensus.cc:2391] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:30.007488 1967 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:30.008154 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.912959 1949 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.010073 1949 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:29.912807 1901 raft_consensus.cc:2747] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:29.932734 1896 consensus_queue.cc:260] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.011312 1896 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:29.926621 1401 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:30.013592 1401 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:29.723456 1240 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:30.014127 1240 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:29.908118 1910 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:29.957152 31803 raft_consensus.cc:1273] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:30.026597 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:30.026809 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 230900us
I20250831 07:59:30.027274 31803 raft_consensus.cc:2391] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:30.027546 31848 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:30.028486 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:29.963905 31699 leader_election.cc:336] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.569s (SENT)
I20250831 07:59:29.963737 1946 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
W20250831 07:59:29.965770 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 201233us
I20250831 07:59:30.036444 1997 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:30.037061 1997 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:30.038959 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:30.040079 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.040688 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:30.041253 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.041476 32470 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:30.042405 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.043797 1949 consensus_queue.cc:1035] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250831 07:59:30.044064 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:29.966996 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.046913 31807 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:30.048038 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.048579 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:30.048871 31807 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
I20250831 07:59:29.945293 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.970516 31775 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:29.974460 1606 raft_consensus.cc:738] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:29.974907 31733 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:29.886152 1827 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:29.990844 1826 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:29.924443 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:30.005213 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 244) took 1898 ms. Trace:
I20250831 07:59:29.924106 1378 consensus_queue.cc:1230] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer a0d845d01c8c4894895f66e3beae298c log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250831 07:59:30.005071 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:30.007892 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.107630 31809 raft_consensus.cc:2391] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:29.994804 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:29.942185 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:30.109110 31726 raft_consensus.cc:1273] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 07:59:29.920459 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.110805 31805 raft_consensus.cc:2408] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250831 07:59:30.018586 31800 raft_consensus.cc:2408] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 07:59:30.019487 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.020305 31730 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:30.111577 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.112531 31804 raft_consensus.cc:2391] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:29.995721 31734 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:30.021456 31727 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:30.023355 31731 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:30.029526 31699 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:30.113969 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 150096us
W20250831 07:59:30.082144 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:29.968526 31849 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:30.115289 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 146851us
I20250831 07:59:30.063063 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:30.116360 32467 raft_consensus.cc:2408] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250831 07:59:30.070218 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.116885 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.116286 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.117327 31803 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 9.
I20250831 07:59:30.091084 31801 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:30.092159 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:30.093293 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 368713us
I20250831 07:59:30.118889 31807 raft_consensus.cc:2391] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:30.094085 32435 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:30.050822 31773 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
W20250831 07:59:30.099001 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 128573us
I20250831 07:59:30.114075 31731 raft_consensus.cc:1238] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:30.116760 31726 raft_consensus.cc:1238] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
W20250831 07:59:30.120687 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 69958.1us
I20250831 07:59:30.121542 31726 raft_consensus.cc:1238] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:30.100345 31732 rpcz_store.cc:276] 0831 07:59:28.106353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:29.573361 (+1467008us) service_pool.cc:225] Handling call
0831 07:59:29.573634 (+ 273us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:30.003957 (+430323us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:30.005021 (+ 1064us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54003e2454. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:30.005164 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":437851008}
I20250831 07:59:30.123420 31731 raft_consensus.cc:1238] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 7. Current term is 8. Ops: []
I20250831 07:59:30.117442 31808 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:30.124743 31807 raft_consensus.cc:1238] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 8. Current term is 9. Ops: []
I20250831 07:59:30.125427 31775 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:30.127279 31807 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:30.127519 31804 raft_consensus.cc:1273] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250831 07:59:30.127543 31775 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:30.130501 31775 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:30.049170 31802 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:30.115023 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.137692 31809 raft_consensus.cc:2391] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:30.131675 31775 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:30.105578 1860 consensus_queue.cc:1046] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20250831 07:59:30.047675 1396 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:30.144527 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 50552.2us
I20250831 07:59:30.137360 1735 raft_consensus.cc:2747] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:30.145112 31775 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:30.095156 1882 raft_consensus.cc:3053] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:30.106082 1901 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:30.116170 1827 raft_consensus.cc:695] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:30.157400 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:30.172734 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 58127.9us
I20250831 07:59:30.126700 1791 raft_consensus.cc:2747] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250831 07:59:30.149376 1867 raft_consensus.cc:2747] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:30.149537 1880 raft_consensus.cc:2802] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:30.142403 1840 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:30.099799 1947 raft_consensus.cc:2747] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:30.045337 1884 consensus_queue.cc:1046] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
I20250831 07:59:30.162767 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.164151 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:30.197055 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:30.196825 32434 leader_election.cc:336] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.517s (SENT)
I20250831 07:59:30.242590 32434 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:30.165742 31775 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:30.187727 31806 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:30.169485 31802 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:30.145036 31734 raft_consensus.cc:1273] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250831 07:59:30.173379 31699 leader_election.cc:336] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.628s (SENT)
I20250831 07:59:30.245661 31699 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:30.246196 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 49459.7us
W20250831 07:59:30.246532 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 73187.6us
I20250831 07:59:30.246568 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:30.175460 31733 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 8.
W20250831 07:59:30.175904 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 133930us
I20250831 07:59:30.147503 1962 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:30.178833 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:30.147713 1871 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:30.170092 1949 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:30.176149 1877 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 07:59:30.276119 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.182307 1378 raft_consensus.cc:2802] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:30.155620 1935 raft_consensus.cc:2802] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 07:59:30.247519 31699 leader_election.cc:336] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.708s (SENT)
I20250831 07:59:30.284056 31699 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:30.285019 1790 raft_consensus.cc:3053] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 LEADER]: Stepping down as leader of term 7
W20250831 07:59:30.285126 31699 leader_election.cc:336] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.734s (SENT)
I20250831 07:59:30.285495 1790 raft_consensus.cc:738] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:30.285626 31699 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:30.197252 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.286373 1790 consensus_queue.cc:260] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:30.286929 31699 leader_election.cc:336] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.747s (SENT)
I20250831 07:59:30.287348 31699 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:30.164682 1882 raft_consensus.cc:738] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
W20250831 07:59:30.252986 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:30.252627 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 86949.1us
I20250831 07:59:30.288405 1882 consensus_queue.cc:260] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:30.290002 1882 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:30.290267 2018 log.cc:927] Time spent T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.386s user 0.002s sys 0.000s
I20250831 07:59:30.286392 1737 raft_consensus.cc:2747] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:30.253347 31800 raft_consensus.cc:1238] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:30.292376 1897 raft_consensus.cc:3053] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:30.292869 1897 raft_consensus.cc:738] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:30.265249 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:30.294059 31882 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:30.264566 31808 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:30.184584 1880 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:30.300519 1880 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:30.304840 1737 raft_consensus.cc:2747] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:30.305784 1737 raft_consensus.cc:2747] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:30.306993 1740 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:30.307734 1740 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.308159 1880 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:30.171027 1965 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:30.309119 1965 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.311319 1965 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:30.311746 1880 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:30.313515 32435 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:30.288852 31775 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:30.315527 31775 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:30.316489 1947 raft_consensus.cc:2802] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:30.316859 1947 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:30.317188 1947 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:30.317289 31882 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:30.321528 1897 consensus_queue.cc:260] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:30.324016 1840 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:30.287595 1790 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:30.325194 1867 consensus_queue.cc:1046] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:30.281524 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:30.326745 31875 raft_consensus.cc:2466] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 2.
I20250831 07:59:30.313011 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:30.323762 1965 consensus_queue.cc:1035] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:59:30.328120 1947 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:30.330446 1947 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:30.311424 1740 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:30.306331 31699 leader_election.cc:336] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.777s (SENT)
I20250831 07:59:30.339661 31699 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:30.342195 1740 raft_consensus.cc:2747] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:30.347469 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:30.347407 31699 leader_election.cc:336] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.827s (SENT)
I20250831 07:59:30.354151 31733 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:30.282650 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:30.244979 32464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47886 (request call id 455) took 1548 ms. Trace:
I20250831 07:59:30.251768 1385 raft_consensus.cc:3053] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:30.270591 1401 raft_consensus.cc:2747] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250831 07:59:30.356292 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 108059us
W20250831 07:59:30.357406 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 44026us
I20250831 07:59:30.251576 1860 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:30.268002 1966 raft_consensus.cc:3053] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:30.354898 31879 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:30.251538 1962 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:30.169582 1396 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.266217 1946 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:30.371945 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:30.376857 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.177870 1901 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:30.283241 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.269827 1949 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:30.278249 1877 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:30.417060 32467 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Advancing to term 8
W20250831 07:59:30.393301 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 74802.2us
I20250831 07:59:30.283803 1378 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:30.391753 1966 raft_consensus.cc:738] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:30.172987 1606 consensus_queue.cc:260] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.353914 1935 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:30.388350 1840 raft_consensus.cc:695] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:30.370540 2000 raft_consensus.cc:2802] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:30.164033 1804 consensus_queue.cc:260] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:30.369890 1385 raft_consensus.cc:738] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:30.369259 1997 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:30.391341 2015 tablet_replica.cc:331] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c: stopping tablet replica
I20250831 07:59:30.398298 1962 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:30.419651 31882 raft_consensus.cc:1811] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Handling vote request from an unknown peer a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:30.358389 1880 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:30.420660 1378 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:30.396800 1860 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:30.413758 1910 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:30.418445 1949 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:30.412010 1882 consensus_queue.cc:1046] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20250831 07:59:30.418382 1877 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:30.419755 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:30.376504 1827 consensus_queue.cc:237] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.430733 1935 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:30.428637 2034 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:30.416548 1624 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:30.436554 2000 raft_consensus.cc:695] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:30.432353 1871 consensus_queue.cc:1162] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7512ba2ed3a94982a22a75f51a647e65" responder_term: 6 status { last_received { term: 5 index: 8 } last_committed_idx: 8 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250831 07:59:30.451696 1897 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:30.459347 1877 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:30.462790 1396 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:30.445044 1880 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:30.471724 1840 consensus_queue.cc:237] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:30.473291 2023 log.cc:927] Time spent T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20250831 07:59:30.477312 1860 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:30.502489 2000 consensus_queue.cc:237] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.479888 1901 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:30.480710 2026 log.cc:927] Time spent T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250831 07:59:30.481496 1804 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:30.506341 1949 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.506906 1877 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:30.485328 2034 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.510646 2034 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:30.501355 1966 consensus_queue.cc:260] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.521574 1966 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:30.374154 31698 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:30.568068 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 193870us
I20250831 07:59:30.352231 31699 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:30.577207 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 158973us
I20250831 07:59:30.578722 31800 raft_consensus.cc:1273] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 07:59:30.583827 31808 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:30.584636 31698 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:30.591014 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.592998 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:30.435256 31882 raft_consensus.cc:2391] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:30.361596 32464 rpcz_store.cc:276] 0831 07:59:28.696403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.710494 (+ 14091us) service_pool.cc:225] Handling call
0831 07:59:28.856619 (+146125us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:28.886617 (+ 29998us) raft_consensus.cc:1532] Early marking committed up to index 7
0831 07:59:29.752040 (+865423us) log.cc:844] Serialized 14 byte log entry
0831 07:59:30.147961 (+395921us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:30.155347 (+ 7386us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:30.169304 (+ 13957us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:30.173181 (+ 3877us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:30.183873 (+ 10692us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":655536,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":55,"raft.queue_time_us":28455,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":6780,"thread_start_us":1070491,"threads_started":2,"wal-append.queue_time_us":659875}
I20250831 07:59:30.461449 32467 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 8.
I20250831 07:59:30.453670 32469 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:30.465528 1385 consensus_queue.cc:260] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:30.482407 1378 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:30.667161 1378 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:30.454887 1240 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:30.632297 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 285211us
I20250831 07:59:30.636059 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:30.668632 31808 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
W20250831 07:59:30.668730 31699 leader_election.cc:336] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.833s (SENT)
I20250831 07:59:30.669334 31699 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:30.670369 1240 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:30.670986 1606 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:30.671870 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.672147 1275 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:30.672211 1995 raft_consensus.cc:2747] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:30.672859 1275 raft_consensus.cc:695] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:59:30.671502 31699 leader_election.cc:336] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.843s (SENT)
I20250831 07:59:30.673684 31699 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:30.674810 1995 raft_consensus.cc:2747] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:30.674389 1275 consensus_queue.cc:237] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:30.675742 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.637560 1901 raft_consensus.cc:2802] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 07:59:30.676545 1901 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:30.676760 31699 leader_election.cc:336] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.939s (SENT)
I20250831 07:59:30.677093 1901 raft_consensus.cc:3058] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:30.677325 31699 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:30.679072 31699 leader_election.cc:336] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.847s (SENT)
I20250831 07:59:30.639124 1949 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:30.680163 1950 raft_consensus.cc:2747] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:30.638236 1826 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.682976 1826 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:30.640128 1855 consensus_queue.cc:1035] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:59:30.683758 31773 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:30.636134 1740 raft_consensus.cc:2747] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:30.642597 1880 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:30.532177 1962 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.650390 1624 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:30.651361 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.432991 32463 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:30.653028 1867 consensus_queue.cc:1035] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:59:30.656046 1396 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:30.656424 1877 consensus_queue.cc:1046] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:59:30.657999 1947 raft_consensus.cc:3053] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:30.684939 1804 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:30.687426 1962 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:30.652390 31698 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:30.679679 31699 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:30.696484 1396 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.696604 1804 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:30.696696 1947 raft_consensus.cc:738] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:30.706897 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 73254.8us
I20250831 07:59:30.705749 1804 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:30.707648 31847 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:30.708381 1877 consensus_queue.cc:1230] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer a0d845d01c8c4894895f66e3beae298c log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 07:59:30.709159 1826 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250831 07:59:30.723351 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.724351 31698 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:30.726365 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.714932 1826 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:30.727305 1826 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:30.729110 1995 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:30.732631 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:30.734629 1624 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:30.735404 1624 raft_consensus.cc:695] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 8 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:30.736893 1624 consensus_queue.cc:237] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.738443 31808 raft_consensus.cc:2466] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:30.740343 1995 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:30.742529 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:30.746850 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:30.747710 1396 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:30.749306 1843 raft_consensus.cc:3053] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:30.749836 1843 raft_consensus.cc:738] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:30.753433 32435 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:30.721307 1740 raft_consensus.cc:2747] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:30.765434 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:30.751343 1843 consensus_queue.cc:260] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:30.768302 1396 raft_consensus.cc:2802] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:30.743764 1901 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.769308 1947 consensus_queue.cc:260] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:30.774147 31726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.001s sys 0.009s
I20250831 07:59:30.776015 1396 raft_consensus.cc:695] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:30.776320 31726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.001s sys 0.009s
I20250831 07:59:30.776569 1843 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:30.777230 1396 consensus_queue.cc:237] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.780164 1947 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:30.777751 1498 raft_consensus.cc:2747] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:30.784394 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.785233 31802 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
I20250831 07:59:30.798053 32435 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:30.795805 31698 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:30.809338 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.809949 32469 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
W20250831 07:59:30.816447 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.786361 1935 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:30.827132 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 55370.7us
W20250831 07:59:30.830091 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.830689 1966 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:30.831162 1966 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:30.832868 31698 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:30.833639 1740 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:30.834077 1740 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:30.834465 1740 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:30.831568 1966 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:30.844518 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:30.845299 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:59:30.845552 31732 raft_consensus.cc:2391] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 8: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:30.848953 1891 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:30.849575 1891 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.854714 1891 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:30.858088 1950 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:30.858731 1950 raft_consensus.cc:695] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 6 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:30.858805 1826 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.859890 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:30.860953 31804 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:30.901463 1901 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:30.907196 32435 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:30.908854 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:30.911412 1606 raft_consensus.cc:2802] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:30.911957 1606 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:30.912362 1606 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:30.912770 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 55946.5us
I20250831 07:59:30.917938 1935 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:30.919389 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 54939.3us
W20250831 07:59:30.919571 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.933943 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:30.859903 1950 consensus_queue.cc:237] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.932767 31726 raft_consensus.cc:1273] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20250831 07:59:30.932904 31725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.809s user 0.006s sys 0.005s
W20250831 07:59:30.939941 32434 leader_election.cc:336] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.835s (SENT)
I20250831 07:59:30.942230 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:30.942610 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:30.942917 31879 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:30.931486 1826 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:30.950088 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:30.955134 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:30.957278 32463 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
W20250831 07:59:30.961006 31725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.944s user 0.007s sys 0.006s
I20250831 07:59:30.964803 31849 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:30.965941 31773 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:30.966573 31849 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:30.967234 1401 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:30.967885 1401 raft_consensus.cc:695] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:30.967491 1740 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.968083 1378 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:30.968664 1378 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:30.969084 1378 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:30.969118 1401 consensus_queue.cc:237] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.969660 1624 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:30.971573 1624 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:30.967649 31849 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:30.977332 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:30.976114 1624 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:30.987287 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 501149us
I20250831 07:59:30.947134 1804 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:30.988535 1949 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:30.995119 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:30.987032 1385 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:31.000785 31882 raft_consensus.cc:3058] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:30.997593 1949 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.006362 1949 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:31.011184 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:31.012307 2058 log.cc:927] Time spent T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.068s user 0.001s sys 0.002s
I20250831 07:59:31.013533 31879 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:31.020445 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1348) took 1194 ms. Trace:
I20250831 07:59:31.032203 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:31.035005 2050 log.cc:927] Time spent T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.080s user 0.000s sys 0.004s
I20250831 07:59:31.038956 1855 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250831 07:59:31.039579 2040 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:31.043926 2040 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:31.045516 2040 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:31.041945 31802 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:31.049556 31698 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:31.053037 1275 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:31.060966 32435 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:31.062103 1740 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:31.076443 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:31.076157 1966 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.028231 31725 rpcz_store.cc:276] 0831 07:59:29.826139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:29.999901 (+173762us) service_pool.cc:225] Handling call
0831 07:59:30.000190 (+ 289us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:30.000379 (+ 189us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 07:59:30.017243 (+ 16864us) log.cc:844] Serialized 14 byte log entry
0831 07:59:30.989036 (+971793us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:30.993260 (+ 4224us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:30.995856 (+ 2596us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:31.007301 (+ 11445us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:31.020392 (+ 13091us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18839,"prepare.queue_time_us":17045,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":7170,"raft.queue_time_us":4842,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":2325,"thread_start_us":944000,"threads_started":1,"wal-append.queue_time_us":747921}
I20250831 07:59:31.054541 1882 consensus_queue.cc:1035] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.018s
I20250831 07:59:31.088539 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:31.089207 31879 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:31.089877 1966 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:31.096805 31879 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:31.108624 31849 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:31.110571 2014 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:31.111208 2014 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.112036 1897 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:31.112323 32463 raft_consensus.cc:1238] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 7. Current term is 8. Ops: []
I20250831 07:59:31.113348 2014 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:31.112843 1897 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.114063 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:31.114786 31800 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:31.086898 1804 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:31.066009 1949 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:31.117344 1949 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:31.117712 1949 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:31.117606 1240 consensus_queue.cc:1046] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250831 07:59:31.118407 1240 consensus_queue.cc:1230] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer 7ecb257174554a26b2c35ad04b0eabae log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250831 07:59:31.118870 1240 raft_consensus.cc:3053] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 07:59:31.119163 1240 raft_consensus.cc:738] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:31.121830 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:31.122440 32469 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:31.072503 1275 raft_consensus.cc:695] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:31.121861 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:31.125088 1275 consensus_queue.cc:237] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.116508 1897 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:31.120110 1240 consensus_queue.cc:260] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.132823 1897 consensus_queue.cc:1035] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 07:59:31.133035 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:31.133960 1240 raft_consensus.cc:3058] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:31.136154 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:31.140837 2055 log.cc:927] Time spent T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20250831 07:59:31.141714 32469 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 8.
I20250831 07:59:31.143915 31773 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:31.144784 1804 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:31.145444 1804 raft_consensus.cc:695] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 8 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:31.146337 1804 consensus_queue.cc:237] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:31.156064 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.161128 1757 consensus_queue.cc:1035] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250831 07:59:31.165937 1378 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:31.168484 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.168747 1378 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:31.168864 1606 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.173631 1995 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:31.178579 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:31.181252 32467 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:31.197945 31698 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:31.212432 31808 raft_consensus.cc:1273] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250831 07:59:31.199891 1826 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:31.224504 31725 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:31.194590 31800 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:31.196007 31882 raft_consensus.cc:2466] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
W20250831 07:59:31.228869 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 99741.2us
I20250831 07:59:31.200068 1624 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.232712 31698 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:31.233703 32435 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
W20250831 07:59:31.234323 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 69981.9us
W20250831 07:59:31.240317 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.220009 1757 consensus_queue.cc:1035] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20250831 07:59:31.242686 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:31.229285 2014 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:31.228808 1826 raft_consensus.cc:695] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:31.219352 1995 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.249028 31808 raft_consensus.cc:1273] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 07:59:31.242166 1950 raft_consensus.cc:2802] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader election won for term 3
I20250831 07:59:31.253695 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:31.260258 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 151787us
I20250831 07:59:31.246820 1966 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:31.247632 1871 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:31.263298 1871 raft_consensus.cc:695] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:31.263406 2015 raft_consensus.cc:2241] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 07:59:31.264633 1624 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:31.264549 1871 consensus_queue.cc:237] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.265144 1606 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:31.263144 1966 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:31.268117 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.269587 1966 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:31.270366 1840 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:31.271325 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:31.271656 1947 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:31.272195 1947 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.273026 31802 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 9.
I20250831 07:59:31.273389 1995 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:31.273988 1947 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:31.275031 2015 raft_consensus.cc:2270] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 07:59:31.270864 1840 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.278200 32435 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:31.279584 1966 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 07:59:31.279636 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:31.280145 1966 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:31.280521 32469 raft_consensus.cc:2366] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8.
I20250831 07:59:31.280570 1966 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Advancing to term 10
W20250831 07:59:31.282140 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.282105 1949 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.260040 1396 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.003s
I20250831 07:59:31.284940 2015 ts_tablet_manager.cc:1905] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 07:59:31.282184 31728 raft_consensus.cc:1238] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:31.285403 2040 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.285825 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:31.287505 2040 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:31.288826 31848 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:31.289567 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:31.296725 31800 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:31.289574 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:31.288856 31727 raft_consensus.cc:1273] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:31.295480 31725 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:31.299005 31732 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 8
W20250831 07:59:31.303493 31699 leader_election.cc:336] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.975s (SENT)
I20250831 07:59:31.304414 31698 leader_election.cc:400] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8.
I20250831 07:59:31.304498 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:31.305935 1826 consensus_queue.cc:237] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.304791 31698 leader_election.cc:403] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
W20250831 07:59:31.310098 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:31.311159 31773 leader_election.cc:336] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.680s (SENT)
I20250831 07:59:31.320052 32469 raft_consensus.cc:1273] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 07:59:31.329556 31699 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:31.330669 1740 raft_consensus.cc:2747] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 07:59:31.331941 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 70683.6us
W20250831 07:59:31.332448 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 43776.3us
I20250831 07:59:31.332886 1624 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:59:31.334525 1624 consensus_queue.cc:1046] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250831 07:59:31.332727 31699 leader_election.cc:336] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.996s (SENT)
I20250831 07:59:31.336035 31699 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:31.336485 31849 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:31.335678 1624 raft_consensus.cc:3053] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:31.347398 31730 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:31.260927 1950 raft_consensus.cc:695] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:59:31.359150 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.366178 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:31.367046 31804 raft_consensus.cc:2408] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 07:59:31.298185 1840 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:31.332742 1897 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:31.367833 31732 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 8.
W20250831 07:59:31.370833 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:31.371464 31699 leader_election.cc:336] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 2.057s (SENT)
I20250831 07:59:31.375931 31631 catalog_manager.cc:5582] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:31.376945 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:31.377836 31809 raft_consensus.cc:2408] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 07:59:31.377895 31631 catalog_manager.cc:5582] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:59:31.381347 31882 raft_consensus.cc:1273] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 07:59:31.381965 1950 consensus_queue.cc:237] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.383740 2059 raft_consensus.cc:2747] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:31.385696 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:31.386886 31806 raft_consensus.cc:2408] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250831 07:59:31.344410 1624 raft_consensus.cc:738] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:31.388811 1624 consensus_queue.cc:260] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.390465 1624 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:31.372570 1995 raft_consensus.cc:2747] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8.
I20250831 07:59:31.392555 2040 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:31.397868 31848 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:31.399986 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 63609.1us
I20250831 07:59:31.400910 2069 raft_consensus.cc:2747] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:31.401015 31849 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:31.401614 2069 consensus_queue.cc:1035] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:59:31.402285 31849 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:31.402479 2014 raft_consensus.cc:2802] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:31.402951 2014 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:31.403061 2069 raft_consensus.cc:2762] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250831 07:59:31.403380 2014 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:31.402679 31848 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:31.403591 31849 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:31.397637 31631 catalog_manager.cc:5582] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:31.418637 31699 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:31.420189 2040 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:31.431083 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:31.420336 1949 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:31.418166 31631 catalog_manager.cc:5582] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 4 to 8, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 8 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:31.392401 1855 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:31.434394 31809 raft_consensus.cc:2391] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:31.437325 31631 catalog_manager.cc:5582] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 3, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 3 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:31.438329 1966 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.445214 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:31.446728 1826 raft_consensus.cc:2747] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:31.450130 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:31.464444 31806 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:31.456190 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 88487.8us
I20250831 07:59:31.459028 31631 catalog_manager.cc:5582] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:31.465731 31804 raft_consensus.cc:2391] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:31.467790 31631 catalog_manager.cc:5582] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:31.468755 31800 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:31.413862 1740 consensus_queue.cc:1035] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
I20250831 07:59:31.466658 31775 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:31.469825 31631 catalog_manager.cc:5582] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250831 07:59:31.472685 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 70133.1us
I20250831 07:59:31.472977 32435 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:31.476580 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:31.467505 1897 raft_consensus.cc:2802] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:31.478617 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:31.477198 31804 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:31.479647 31732 raft_consensus.cc:2391] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:31.456009 1855 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.480363 1897 raft_consensus.cc:695] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:31.481457 1897 consensus_queue.cc:237] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.482321 1855 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:31.476329 31631 catalog_manager.cc:5582] T 5eac48bce84e483582f863716d5c26f9 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 3 to 6, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 6 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:31.480474 1804 raft_consensus.cc:2747] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:31.489241 31730 raft_consensus.cc:1238] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:31.494777 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:31.495440 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:31.477492 1757 raft_consensus.cc:2802] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:31.498203 31631 catalog_manager.cc:5582] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 2 to 4, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4). New cstate: current_term: 4 leader_uuid: "7ecb257174554a26b2c35ad04b0eabae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 07:59:31.499145 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:31.502033 31848 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:31.498701 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:31.483403 2059 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:31.504730 31800 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:31.504432 31731 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:31.484084 1966 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 10 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:31.513156 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 55097.7us
W20250831 07:59:31.515934 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.497886 31729 raft_consensus.cc:1215] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Deduplicated request from leader. Original: 2.1->[3.2-9.4] Dedup: 3.2->[5.3-9.4]
I20250831 07:59:31.518018 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:31.501369 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:31.526512 31727 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 8
I20250831 07:59:31.529351 32435 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:31.531116 31699 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:31.516667 31729 pending_rounds.cc:85] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 2
I20250831 07:59:31.500150 31631 catalog_manager.cc:5582] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae reported cstate change: term changed from 1 to 4, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 4 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } } }
I20250831 07:59:31.532418 31729 pending_rounds.cc:107] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250831 07:59:31.537812 31729 raft_consensus.cc:2887] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250831 07:59:31.539204 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 53570.3us
I20250831 07:59:31.543740 32467 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250831 07:59:31.545369 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.539070 2059 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.548779 2040 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 07:59:31.529801 1757 raft_consensus.cc:695] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:31.552589 32434 leader_election.cc:336] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 3.161s (SENT)
W20250831 07:59:31.554725 31773 leader_election.cc:336] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.566s (SENT)
I20250831 07:59:31.544313 1966 raft_consensus.cc:2802] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:31.561843 31773 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:31.557253 2059 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:31.551144 1378 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:31.569317 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:31.569653 1378 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.570542 1240 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:31.571086 1240 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.550801 1275 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20250831 07:59:31.573534 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:31.574286 31882 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 3.
W20250831 07:59:31.567862 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 67243.8us
I20250831 07:59:31.572877 1240 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:31.576905 31698 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:31.578632 31806 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:31.581743 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:31.565466 1966 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:31.571739 1378 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:31.619127 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 69864.5us
W20250831 07:59:31.619455 2078 log.cc:927] Time spent T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
W20250831 07:59:31.621286 2080 log.cc:927] Time spent T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.100s user 0.001s sys 0.001s
I20250831 07:59:31.596877 1966 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:31.635519 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:31.662271 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.604535 1757 consensus_queue.cc:237] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.652400 31848 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:31.654942 31804 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 10.
I20250831 07:59:31.580955 1950 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:31.664201 1378 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:31.595499 1275 raft_consensus.cc:2802] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:31.662628 32467 raft_consensus.cc:3053] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:31.665222 32435 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:31.671459 32467 raft_consensus.cc:738] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:31.662993 2074 log.cc:927] Time spent T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.211s user 0.002s sys 0.006s
I20250831 07:59:31.679654 31882 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250831 07:59:31.672614 32435 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
W20250831 07:59:31.684724 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 88277.7us
I20250831 07:59:31.685005 1385 raft_consensus.cc:2802] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:31.685511 1385 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:31.685896 1385 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:31.686787 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:31.687598 31875 raft_consensus.cc:2391] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:31.690897 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:31.667246 1840 raft_consensus.cc:2747] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250831 07:59:31.691972 31881 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 7: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:31.675215 1949 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:31.692798 1949 raft_consensus.cc:695] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:31.670604 2014 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.693316 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:31.693724 1949 consensus_queue.cc:237] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.694948 2014 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:31.695204 2059 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:31.695963 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 141302us
I20250831 07:59:31.696524 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:31.696957 31773 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.638s (SENT) [suppressed 50 similar messages]
I20250831 07:59:31.697351 31882 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:31.694103 31880 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:31.676816 32467 consensus_queue.cc:260] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:31.710208 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 135273us
I20250831 07:59:31.712046 32467 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:31.671377 1275 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:31.723527 31882 raft_consensus.cc:1238] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20250831 07:59:31.723942 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 47681.7us
W20250831 07:59:31.725549 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 131743us
I20250831 07:59:31.708375 1396 raft_consensus.cc:2802] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 10 FOLLOWER]: Leader election won for term 10
W20250831 07:59:31.733067 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.725607 1275 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:31.744088 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:31.745187 31775 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:31.746028 31775 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:31.746111 1867 consensus_queue.cc:1035] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20250831 07:59:31.746191 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.679924 1378 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.711010 1740 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20250831 07:59:31.755744 32467 raft_consensus.cc:1273] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20250831 07:59:31.756404 32467 pending_rounds.cc:85] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: Aborting all ops after (but not including) 6
I20250831 07:59:31.756697 32467 pending_rounds.cc:107] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20250831 07:59:31.756989 32467 raft_consensus.cc:2887] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250831 07:59:31.776211 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 79262.8us
I20250831 07:59:31.720692 2059 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.783128 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:31.796834 32467 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 07:59:31.800348 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:31.798333 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 63252.4us
W20250831 07:59:31.792280 2071 log.cc:927] Time spent T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.208s user 0.003s sys 0.006s
W20250831 07:59:31.802249 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 68727.8us
I20250831 07:59:31.782723 1947 raft_consensus.cc:2802] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election won for term 4
W20250831 07:59:31.803079 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.792949 1867 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:31.783836 1843 raft_consensus.cc:2802] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
W20250831 07:59:31.804821 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:31.799996 2081 log.cc:927] Time spent T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.141s user 0.000s sys 0.007s
I20250831 07:59:31.793552 1396 raft_consensus.cc:695] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [term 10 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:31.805025 2079 log.cc:927] Time spent T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.176s user 0.007s sys 0.000s
I20250831 07:59:31.816354 31877 raft_consensus.cc:1238] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
W20250831 07:59:31.817363 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.803536 1240 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:31.805388 1962 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:31.825198 31699 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:31.814731 1867 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.828934 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:31.806818 1826 consensus_queue.cc:1035] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
W20250831 07:59:31.873034 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 125862us
I20250831 07:59:31.829206 1947 raft_consensus.cc:695] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:31.845309 2014 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:31.824021 1935 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20250831 07:59:31.899137 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:31.869072 1740 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:31.829679 1962 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.904242 31727 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 8.
I20250831 07:59:31.844892 1240 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.916535 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:31.918375 31883 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
W20250831 07:59:31.920634 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 99669.2us
I20250831 07:59:31.901749 2089 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:31.933589 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:31.935678 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:31.898013 2014 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.910455 2015 ts_tablet_manager.cc:1918] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 07:59:31.951010 2059 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:31.975708 31773 leader_election.cc:336] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.730s (SENT)
I20250831 07:59:31.976254 31773 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:31.976862 2088 log.cc:927] Time spent T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.133s user 0.001s sys 0.000s
I20250831 07:59:31.944312 1843 raft_consensus.cc:695] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:31.976943 1385 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.953835 1947 consensus_queue.cc:237] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.951033 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:31.884456 1396 consensus_queue.cc:237] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:31.996480 1843 consensus_queue.cc:237] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:31.997815 1385 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:31.991946 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:32.000012 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:32.000293 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:31.954885 1378 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:32.002900 31882 raft_consensus.cc:2408] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250831 07:59:32.005391 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 72553.1us
W20250831 07:59:32.006392 31773 leader_election.cc:336] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 2.012s (SENT)
I20250831 07:59:31.955150 2089 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:32.003960 2015 log.cc:1199] T 26525016efb247c3b1c2bf7fec0e1658 P a0d845d01c8c4894895f66e3beae298c: Deleting WAL directory at /tmp/dist-test-taskPkGbrE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756627026548763-30564-0/minicluster-data/ts-0-root/wals/26525016efb247c3b1c2bf7fec0e1658
I20250831 07:59:32.000871 2059 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:32.004228 1867 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:32.029544 31699 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:32.032270 2094 log.cc:927] Time spent T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20250831 07:59:32.032492 31880 raft_consensus.cc:1238] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:32.041242 31883 raft_consensus.cc:604] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:32.041844 31883 raft_consensus.cc:612] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250831 07:59:32.039654 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 162939us
I20250831 07:59:32.040846 1826 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:32.036727 2059 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:32.102664 32434 leader_election.cc:336] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.647s (SENT)
I20250831 07:59:32.114459 31879 raft_consensus.cc:1238] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:32.095940 1240 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:32.100749 1962 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:32.119784 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:32.086720 1826 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:32.127370 31880 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:32.148042 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:32.150156 31881 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:32.115487 2015 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57666 (request call id 529) took 3612 ms. Trace:
I20250831 07:59:32.117857 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:32.162263 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:32.126374 1378 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:32.147403 1962 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:32.197023 31881 raft_consensus.cc:1238] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
W20250831 07:59:32.199554 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:32.199033 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:32.157378 2014 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:32.204711 31879 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 9.
W20250831 07:59:32.203918 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 105494us
W20250831 07:59:32.206398 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 133355us
W20250831 07:59:32.212364 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:32.212944 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:32.213760 31727 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:32.197680 32463 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:32.214589 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:32.161198 2104 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:32.192349 2112 raft_consensus.cc:2747] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:32.223989 31881 raft_consensus.cc:1238] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:32.227875 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:32.242846 31875 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:32.244834 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:32.245882 31881 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 7: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
W20250831 07:59:32.252277 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:32.243937 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:32.195525 1275 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:32.205317 1962 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:32.326807 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:32.326984 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 308654us
I20250831 07:59:32.327716 31875 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 9.
I20250831 07:59:32.329751 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:32.326879 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:32.330696 31882 raft_consensus.cc:2408] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250831 07:59:32.328032 31729 raft_consensus.cc:1273] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:32.327631 32469 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250831 07:59:32.334309 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:32.334472 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:32.331842 31775 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:32.335340 31881 raft_consensus.cc:2391] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:32.339293 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:32.344036 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 302497us
I20250831 07:59:32.351048 31874 raft_consensus.cc:2391] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:32.327551 2093 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:32.352121 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:32.355362 2082 log.cc:927] Time spent T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
I20250831 07:59:32.351572 1843 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:32.356758 1843 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:32.357177 1843 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:32.358011 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:32.357275 2109 log.cc:927] Time spent T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.156s user 0.000s sys 0.000s
I20250831 07:59:32.227411 2089 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:32.371721 31699 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:32.303156 2106 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:32.380127 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:32.386499 31880 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:32.353989 1826 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:32.320448 1378 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:32.411862 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:32.405588 31876 raft_consensus.cc:1238] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:32.424242 31874 raft_consensus.cc:2374] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5. Re-sending same reply.
I20250831 07:59:32.160984 2015 rpcz_store.cc:276] 0831 07:59:28.539347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:28.539628 (+ 281us) service_pool.cc:225] Handling call
0831 07:59:30.180578 (+1640950us) ts_tablet_manager.cc:1145] Deleting tablet 26525016efb247c3b1c2bf7fec0e1658
0831 07:59:30.265822 (+ 85244us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0831 07:59:31.545875 (+1280053us) tablet_metadata.cc:641] Metadata flushed
0831 07:59:31.877493 (+331618us) tablet_metadata.cc:641] Metadata flushed
0831 07:59:32.115349 (+237856us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":825202,"tablet-delete.queue_time_us":484766,"thread_start_us":368391,"threads_started":1}
I20250831 07:59:32.382462 1867 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:32.422134 2105 log.cc:927] Time spent T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.221s user 0.000s sys 0.001s
W20250831 07:59:32.440553 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:32.441849 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:32.371803 1401 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20250831 07:59:32.445770 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:32.422335 2059 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:32.447732 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:32.368566 2093 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:32.450356 1966 consensus_meta.cc:287] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.787s user 0.009s sys 0.005s
W20250831 07:59:32.450943 2102 log.cc:927] Time spent T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.284s user 0.000s sys 0.001s
I20250831 07:59:32.450781 1966 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:32.452077 2093 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:32.452209 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:32.384920 2106 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:32.457216 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 106441us
I20250831 07:59:32.465945 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:32.471184 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:32.472126 31874 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
W20250831 07:59:32.475006 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:32.454952 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 106547us
I20250831 07:59:32.446959 1867 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:32.512418 31880 raft_consensus.cc:1273] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250831 07:59:32.512773 2127 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:32.513314 2127 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:32.514050 31618 catalog_manager.cc:4928] TS a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): tablet 26525016efb247c3b1c2bf7fec0e1658 (table test-workload [id=ffdd45d780464eb78bd0f5ef742a4263]) successfully deleted
W20250831 07:59:32.515734 2085 log.cc:927] Time spent T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.636s user 0.002s sys 0.002s
I20250831 07:59:32.516968 31879 raft_consensus.cc:1273] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:32.532745 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:32.544649 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:32.533696 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:32.610916 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:32.666930 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:32.667845 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:32.538036 2134 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:32.512522 2116 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:32.676383 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:32.635474 2123 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:32.635939 2114 raft_consensus.cc:2747] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:32.679962 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:32.685578 31734 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:32.648147 2127 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:32.688118 31775 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:32.689095 2087 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:32.688091 2123 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:32.682475 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:32.552019 1275 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:32.689656 2087 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:32.690030 31772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.017s sys 0.013s
W20250831 07:59:32.721542 31772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s user 0.018s sys 0.013s
I20250831 07:59:32.690636 2134 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:32.691720 2121 log.cc:927] Time spent T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.332s user 0.000s sys 0.002s
W20250831 07:59:32.693499 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:32.685454 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:32.692435 31876 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:32.664783 1962 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:32.759253 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 67626.5us
I20250831 07:59:32.760845 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:32.709677 2116 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:32.794796 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 107044us
W20250831 07:59:32.828954 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:32.832353 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:32.832886 31883 raft_consensus.cc:2391] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:32.836820 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:32.837482 31734 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:32.837610 2123 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:32.766942 1840 raft_consensus.cc:2802] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:32.838552 1840 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:32.838948 1840 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:32.839393 2130 log.cc:927] Time spent T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20250831 07:59:32.842087 1804 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:32.842592 1804 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:32.843561 31773 leader_election.cc:336] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 2.960s (SENT)
I20250831 07:59:32.844173 31734 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd"
dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
mode: GRACEFUL
new_leader_uuid: "8247a38b550649aa9cd6e7668dcaec10"
from {username='slave'} at 127.0.0.1:48724
W20250831 07:59:32.845171 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 218040us
I20250831 07:59:32.847903 31734 raft_consensus.cc:604] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Received request to transfer leadership to TS 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:32.848304 31734 raft_consensus.cc:612] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250831 07:59:32.848457 2138 log.cc:927] Time spent T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.185s user 0.001s sys 0.000s
I20250831 07:59:32.848601 31699 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:32.831696 2122 log.cc:927] Time spent T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.391s user 0.001s sys 0.000s
I20250831 07:59:32.846217 31725 raft_consensus.cc:1273] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20250831 07:59:32.850512 2144 log.cc:927] Time spent T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20250831 07:59:32.836110 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:32.855206 31877 raft_consensus.cc:2374] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6. Re-sending same reply.
I20250831 07:59:32.871577 31698 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:32.873471 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:32.767259 1826 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:32.787928 1378 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:32.876901 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:32.789016 2142 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:32.878248 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:32.884541 31883 raft_consensus.cc:2391] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:32.881192 31877 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:32.888676 2134 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:32.889320 31699 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:32.890798 2141 log.cc:927] Time spent T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20250831 07:59:32.890998 31699 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:32.894985 31775 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:32.899729 1867 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:59:32.907379 1867 raft_consensus.cc:2802] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:32.907752 1867 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:32.908088 1867 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:32.911468 1804 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:32.900499 31734 raft_consensus.cc:1273] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:59:32.880709 1962 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:32.912832 1962 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:32.914956 1962 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:32.917068 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:32.873054 2123 raft_consensus.cc:2747] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:32.923839 31876 raft_consensus.cc:1238] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
W20250831 07:59:32.924430 2139 log.cc:927] Time spent T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.261s user 0.009s sys 0.005s
W20250831 07:59:32.921923 31773 leader_election.cc:336] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 2.317s (SENT)
I20250831 07:59:32.925798 31773 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:32.926885 31882 raft_consensus.cc:1273] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:32.926846 1962 raft_consensus.cc:2747] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250831 07:59:32.927074 2113 log.cc:927] Time spent T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.713s user 0.000s sys 0.002s
I20250831 07:59:32.928416 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:32.928783 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:32.929077 31881 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
W20250831 07:59:32.906733 1740 consensus_meta.cc:287] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.990s user 0.006s sys 0.006s
I20250831 07:59:32.928373 31874 raft_consensus.cc:1238] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:32.929160 31879 raft_consensus.cc:1273] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20250831 07:59:32.934389 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:32.939699 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 68230.2us
I20250831 07:59:32.904909 2137 raft_consensus.cc:2747] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 07:59:33.015036 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 83370.6us
W20250831 07:59:33.016599 31773 leader_election.cc:336] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 2.402s (SENT)
W20250831 07:59:33.017215 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 118000us
I20250831 07:59:32.925700 1966 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:32.930493 1740 raft_consensus.cc:2747] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:32.930835 2106 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:33.033689 2146 consensus_queue.cc:1035] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20250831 07:59:32.967481 2134 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250831 07:59:33.075551 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:33.088675 32434 leader_election.cc:336] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 2.820s (SENT)
I20250831 07:59:33.048293 2134 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:33.089421 32434 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:33.032109 2136 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:33.096534 2096 raft_consensus.cc:2747] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:33.089550 2134 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:33.100466 2087 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:33.034801 2147 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:33.108114 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:33.143730 1740 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:33.145017 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:33.144582 1740 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:33.147279 1740 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:33.157713 1867 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:33.171182 1867 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:33.203971 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:33.165076 1843 consensus_meta.cc:287] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.807s user 0.003s sys 0.008s
I20250831 07:59:33.067049 1378 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:32.950471 2142 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:32.893775 1826 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:33.265048 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 374147us
W20250831 07:59:33.266098 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 177514us
W20250831 07:59:33.273367 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:33.264807 31882 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
W20250831 07:59:33.297894 32434 leader_election.cc:336] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 2.187s (SENT)
I20250831 07:59:33.298008 31882 raft_consensus.cc:604] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:33.295594 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.298569 31882 raft_consensus.cc:612] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:59:33.298427 32434 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:33.258162 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 241575us
I20250831 07:59:33.296212 31877 raft_consensus.cc:1273] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:33.219756 31808 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250831 07:59:33.305728 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:33.309293 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:33.309840 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:33.310179 31877 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:33.311031 31773 leader_election.cc:336] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 2.216s (SENT)
I20250831 07:59:33.309567 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.312279 31802 raft_consensus.cc:1273] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:33.312979 31874 raft_consensus.cc:2391] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:33.314898 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:33.316864 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.317659 31802 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 07:59:33.318341 32471 raft_consensus.cc:2366] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:33.318511 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:33.319028 31808 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
I20250831 07:59:33.266480 1740 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:33.158305 2147 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:33.323675 31699 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:33.307265 1275 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:33.313545 1966 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:33.325862 2135 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:33.326517 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:33.304456 2134 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:33.327704 1840 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:33.328918 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:33.303933 2127 consensus_queue.cc:1035] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20250831 07:59:33.331027 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.366979 31880 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:33.331967 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.332726 31878 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:33.332727 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.368530 31875 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:33.333762 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.369381 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.370079 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.370474 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:33.370899 31879 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:33.371605 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:33.372378 31875 raft_consensus.cc:2391] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:33.310107 2114 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:33.334846 1826 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:33.373211 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.340809 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:33.342586 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:33.374157 31874 raft_consensus.cc:2391] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 4: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:33.374400 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:33.352175 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:33.354151 1275 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:33.375272 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.376032 31874 raft_consensus.cc:2391] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 8: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:33.376549 1275 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:33.362402 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:33.376734 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.304214 1378 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:33.378190 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.379164 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.379287 1378 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:33.304332 1843 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:33.363267 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:33.365324 31728 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 07:59:33.380087 31874 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee"
dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:50826
I20250831 07:59:33.371994 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.380614 31874 raft_consensus.cc:604] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:33.380985 31874 raft_consensus.cc:612] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:59:33.380913 31879 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 3.
I20250831 07:59:33.381135 1843 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:33.335238 1740 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:33.382404 31874 raft_consensus.cc:3053] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:33.382709 31874 raft_consensus.cc:738] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:33.383407 31874 consensus_queue.cc:260] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:33.336978 2150 consensus_queue.cc:1035] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20250831 07:59:33.372965 2114 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:33.383526 31879 raft_consensus.cc:1273] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 07:59:33.384696 31874 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:33.385977 2114 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:33.388785 31733 raft_consensus.cc:2391] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 8: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
W20250831 07:59:33.402442 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:33.383790 1740 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:33.407938 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.409683 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:33.410460 31808 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:33.357652 1995 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 07:59:33.414206 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.415464 31875 raft_consensus.cc:2391] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:33.416718 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.416842 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:33.408823 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250831 07:59:33.426626 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:33.427666 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 120972us
W20250831 07:59:33.429193 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:33.432586 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:33.433431 31732 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:33.438635 31874 raft_consensus.cc:1273] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20250831 07:59:33.439731 31874 pending_rounds.cc:85] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10: Aborting all ops after (but not including) 4
I20250831 07:59:33.440248 31874 pending_rounds.cc:107] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20250831 07:59:33.422528 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:59:33.440711 31874 raft_consensus.cc:2887] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:33.404704 1378 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:33.447893 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 133480us
I20250831 07:59:33.448452 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:33.444824 31731 raft_consensus.cc:1238] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:33.445546 31880 raft_consensus.cc:2391] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
W20250831 07:59:33.425163 2152 log.cc:927] Time spent T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250831 07:59:33.419131 2136 raft_consensus.cc:2747] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:33.459059 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:33.459592 31734 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 6
I20250831 07:59:33.418490 1995 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:33.461028 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:33.465277 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:33.465919 2116 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:33.468765 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.469282 31775 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:33.469463 31874 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:33.364420 2135 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:33.471658 32467 raft_consensus.cc:1273] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250831 07:59:33.476059 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:33.477118 31882 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:33.478025 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:33.478505 31879 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:33.479264 31699 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:33.453253 2131 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:33.486032 1840 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:33.488636 2134 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:33.458715 2132 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:33.496304 2132 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:33.497648 31880 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:33.498361 2132 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:33.501576 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.505254 2147 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:33.506436 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:59:33.507572 31699 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:33.456801 1378 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:33.471334 1995 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:33.509115 31699 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:33.532665 1740 raft_consensus.cc:2747] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 07:59:33.534396 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 65207.2us
I20250831 07:59:33.534323 31699 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:33.511152 1378 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:33.536453 2129 raft_consensus.cc:2747] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:33.511361 2131 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:33.537606 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 364) took 1546 ms. Trace:
I20250831 07:59:33.538781 2131 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:33.538379 2129 consensus_queue.cc:1046] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:59:33.514030 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.547715 31698 leader_election.cc:400] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:33.548054 31698 leader_election.cc:403] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250831 07:59:33.548003 31876 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:33.548897 31734 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:33.530586 31883 raft_consensus.cc:1238] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:33.550375 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.551111 31883 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:33.552162 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:33.552796 31883 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:33.515532 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:33.530882 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:33.542409 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:33.543612 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:33.537976 31804 rpcz_store.cc:276] 0831 07:59:31.990950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:31.991111 (+ 161us) service_pool.cc:225] Handling call
0831 07:59:32.002024 (+ 10913us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:32.002230 (+ 206us) raft_consensus.cc:1532] Early marking committed up to index 8
0831 07:59:32.068301 (+ 66071us) log.cc:844] Serialized 14 byte log entry
0831 07:59:32.068735 (+ 434us) log.cc:844] Serialized 14 byte log entry
0831 07:59:33.500232 (+1431497us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b18002b0600. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce501365a 00007fdcee4e4409 00007fdce7c57ce0 00007fdce7c4b323 00007fdce7c40d8a 00007fdce7c412da 00007fdce80079da 00007fdce801706f 00007fdce8016fe8
0831 07:59:33.537255 (+ 37023us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:33.537311 (+ 56us) raft_consensus.cc:1643] Marking committed up to 8
0831 07:59:33.537362 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:33.537426 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:33.537530 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39943,"prepare.queue_time_us":180276,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":93,"raft.queue_time_us":42418,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":5277,"spinlock_wait_cycles":6444416,"thread_start_us":65700,"threads_started":1}
I20250831 07:59:33.516271 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:33.557055 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:33.546399 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:33.514377 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1382) took 1305 ms. Trace:
I20250831 07:59:33.555747 31808 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
I20250831 07:59:33.557818 31729 rpcz_store.cc:276] 0831 07:59:32.211626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:32.211785 (+ 159us) service_pool.cc:225] Handling call
0831 07:59:32.220861 (+ 9076us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:32.332428 (+111567us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:33.514129 (+1181701us) spinlock_profiling.cc:243] Waited 82.4 ms on lock 0x7b54002eb8d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:33.514300 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":189423360}
I20250831 07:59:33.572136 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:33.571920 32435 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:33.578490 31727 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:33.569890 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:33.510448 2149 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975: Append to log took a long time: real 0.645s user 0.002s sys 0.002s
I20250831 07:59:33.570706 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:33.587704 32469 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:33.588267 31775 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:33.569262 31877 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:33.583653 31732 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
W20250831 07:59:33.586565 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:33.607525 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.608232 31876 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:33.609997 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:33.612502 31698 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:33.614183 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.615124 31880 raft_consensus.cc:2366] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4.
I20250831 07:59:33.531656 2135 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:33.615754 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:33.547547 2134 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:33.620309 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 86054us
I20250831 07:59:33.530359 2142 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:33.550904 2136 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:33.586719 1740 raft_consensus.cc:3058] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 7
I20250831 07:59:33.625046 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:33.638547 31773 leader_election.cc:336] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.638s (SENT)
I20250831 07:59:33.511426 2059 consensus_queue.cc:1035] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.110s
I20250831 07:59:33.630630 31775 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:33.630525 1843 consensus_queue.cc:1035] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 07:59:33.631481 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:33.607537 2087 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:33.586489 2092 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20250831 07:59:33.643601 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.659737 31883 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:33.660493 31880 raft_consensus.cc:2391] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 3: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:33.661605 31876 raft_consensus.cc:2408] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20250831 07:59:33.620803 1966 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 07:59:33.636595 31878 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
I20250831 07:59:33.674363 31878 raft_consensus.cc:1238] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:33.674495 31727 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:33.547513 2125 consensus_queue.cc:1035] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
W20250831 07:59:33.668352 31773 leader_election.cc:336] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.675s (SENT)
I20250831 07:59:33.677158 31773 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:33.621013 2099 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:33.623190 1950 raft_consensus.cc:2802] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:33.613859 1897 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:33.670887 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.671140 31698 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:33.633390 1962 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:33.620507 2112 consensus_queue.cc:1035] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20250831 07:59:33.679150 31877 raft_consensus.cc:1273] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250831 07:59:33.691192 31775 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:33.674801 32434 leader_election.cc:336] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.753s (SENT)
I20250831 07:59:33.607277 2096 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20250831 07:59:33.700819 31883 raft_consensus.cc:2408] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20250831 07:59:33.702339 31879 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:33.709623 31878 raft_consensus.cc:1238] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 8. Ops: [6.7-6.7]
I20250831 07:59:33.713923 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.714995 31879 raft_consensus.cc:2408] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250831 07:59:33.669420 2132 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:33.719180 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.732745 31880 raft_consensus.cc:2408] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
W20250831 07:59:33.720605 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 49541.6us
I20250831 07:59:33.690382 2142 raft_consensus.cc:2802] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
W20250831 07:59:33.742221 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 103368us
I20250831 07:59:33.748407 31882 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:33.724576 1935 raft_consensus.cc:2802] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:33.676584 2087 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:33.756738 31699 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:33.720213 1950 raft_consensus.cc:695] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:33.727777 2132 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:33.769762 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:33.771096 31876 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:33.738597 32464 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:33.738121 2147 raft_consensus.cc:2747] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:33.739087 1962 raft_consensus.cc:695] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:33.737906 2135 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election won for term 7
W20250831 07:59:33.772289 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 97507.4us
I20250831 07:59:33.752322 2142 raft_consensus.cc:695] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:33.774873 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.704555 2099 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:33.773715 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 201915us
W20250831 07:59:33.816170 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 59517.4us
I20250831 07:59:33.770725 1935 raft_consensus.cc:695] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:33.770198 2127 raft_consensus.cc:2747] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:33.734880 1897 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:33.820186 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.796291 1378 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:33.822000 31879 raft_consensus.cc:1273] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250831 07:59:33.854828 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:33.865821 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:33.776262 2132 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Advancing to term 8
W20250831 07:59:33.866371 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 56840.5us
I20250831 07:59:33.820931 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.894457 31699 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:33.909328 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.992108 31877 raft_consensus.cc:2408] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250831 07:59:33.871385 2135 raft_consensus.cc:695] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:33.994347 2135 consensus_queue.cc:237] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:33.972728 1962 consensus_queue.cc:237] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:33.997892 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:33.908732 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:33.937747 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.896211 32435 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
W20250831 07:59:34.002328 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 106199us
I20250831 07:59:33.992218 31882 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 9.
I20250831 07:59:34.002414 31880 raft_consensus.cc:2366] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4.
W20250831 07:59:34.017261 32434 leader_election.cc:336] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.574s (SENT)
I20250831 07:59:34.020781 32434 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:34.034749 32467 raft_consensus.cc:3053] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Stepping down as leader of term 2
I20250831 07:59:34.035296 32467 raft_consensus.cc:738] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:34.015551 31876 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:33.895442 2096 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:34.040534 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:34.041522 32467 consensus_queue.cc:260] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:33.909555 2137 raft_consensus.cc:2762] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250831 07:59:34.048347 32467 raft_consensus.cc:3058] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 2 FOLLOWER]: Advancing to term 3
I20250831 07:59:34.044765 31883 raft_consensus.cc:1273] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250831 07:59:34.049299 31876 raft_consensus.cc:1273] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20250831 07:59:34.049427 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:34.056404 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:34.059396 31698 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:34.060407 1275 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:34.060878 1275 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:34.061280 1275 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:34.062750 31698 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:34.045190 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:34.046895 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:34.064520 31773 leader_election.cc:336] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.724s (SENT)
I20250831 07:59:34.064991 31773 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:34.064285 31880 raft_consensus.cc:3058] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:34.020454 2166 raft_consensus.cc:2802] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader election won for term 7
W20250831 07:59:34.078112 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:34.036160 1966 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:33.912153 2131 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.136s
I20250831 07:59:34.087877 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:33.938192 1378 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:34.043972 2127 raft_consensus.cc:2762] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20250831 07:59:34.091801 31883 raft_consensus.cc:2408] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250831 07:59:33.959184 1950 consensus_queue.cc:237] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:34.106866 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:34.153651 2087 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:34.154127 31875 raft_consensus.cc:2374] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5. Re-sending same reply.
I20250831 07:59:34.154229 2166 raft_consensus.cc:695] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 7 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:34.155234 2166 consensus_queue.cc:237] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:34.156735 2162 log.cc:927] Time spent T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20250831 07:59:34.157943 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.158576 31726 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
W20250831 07:59:34.109153 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:34.161964 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 97218.4us
W20250831 07:59:34.164187 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:34.166937 1947 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:34.167128 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 298570us
I20250831 07:59:34.167498 1947 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:33.959326 1935 consensus_queue.cc:237] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:34.169095 31699 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:34.002172 2142 consensus_queue.cc:237] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:34.186280 1275 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:34.077817 2124 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250831 07:59:34.191414 31849 leader_election.cc:336] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.638s (SENT)
W20250831 07:59:34.192536 32434 leader_election.cc:336] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.920s (SENT)
I20250831 07:59:34.193128 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:34.198729 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:34.099292 2099 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:34.201279 31849 leader_election.cc:336] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.683s (SENT)
I20250831 07:59:34.108594 2100 raft_consensus.cc:2747] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:34.205274 31775 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:34.198048 2114 raft_consensus.cc:2747] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:34.216445 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:34.226060 1275 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:34.055097 2096 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:34.229560 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:34.229455 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:34.230589 31876 raft_consensus.cc:2408] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
W20250831 07:59:34.285507 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 116504us
I20250831 07:59:34.286453 31699 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:34.297694 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 104915us
W20250831 07:59:34.301654 32434 leader_election.cc:336] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 2.888s (SENT)
I20250831 07:59:34.302692 32467 raft_consensus.cc:1273] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20250831 07:59:34.302608 1740 consensus_meta.cc:287] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.641s user 0.009s sys 0.000s
I20250831 07:59:34.303449 1740 raft_consensus.cc:2747] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
W20250831 07:59:34.305904 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 92464.5us
I20250831 07:59:34.307510 31880 raft_consensus.cc:2466] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 10.
I20250831 07:59:34.309099 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:34.309909 31883 raft_consensus.cc:1273] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20250831 07:59:34.310364 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:34.311106 31875 raft_consensus.cc:2374] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 10. Re-sending same reply.
W20250831 07:59:34.312721 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 224033us
I20250831 07:59:34.317085 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:34.318169 31876 raft_consensus.cc:2408] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20250831 07:59:34.337356 31849 leader_election.cc:336] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.791s (SENT)
I20250831 07:59:34.338302 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:34.339035 32467 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:34.326742 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:34.313555 1995 consensus_meta.cc:287] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.789s user 0.011s sys 0.000s
W20250831 07:59:34.344514 32436 leader_election.cc:336] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.631s (SENT)
I20250831 07:59:34.345664 31731 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:34.340252 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:34.349522 31880 raft_consensus.cc:2374] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 10. Re-sending same reply.
I20250831 07:59:34.350419 31775 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:34.363237 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 76876.5us
I20250831 07:59:34.365936 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:34.344944 2059 raft_consensus.cc:2747] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:34.361325 1867 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 07:59:34.336751 2127 consensus_queue.cc:1035] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20250831 07:59:34.383615 2132 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:34.391530 2132 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:34.371618 2099 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:34.360342 1995 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:34.424731 31802 raft_consensus.cc:1273] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250831 07:59:34.424984 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:34.441370 2177 log.cc:927] Time spent T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20250831 07:59:34.383766 1867 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:34.443023 1867 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Advancing to term 9
W20250831 07:59:34.448949 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:34.451854 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:34.457442 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:34.459185 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 94892us
I20250831 07:59:34.475206 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:34.475577 32467 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:34.476066 31808 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:34.476881 31698 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:34.478132 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:34.478263 31882 raft_consensus.cc:1273] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20250831 07:59:34.478668 31882 pending_rounds.cc:85] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10: Aborting all ops after (but not including) 5
I20250831 07:59:34.478900 31882 pending_rounds.cc:107] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20250831 07:59:34.479107 31882 raft_consensus.cc:2887] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250831 07:59:34.479236 32464 consensus_meta.cc:287] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.628s user 0.009s sys 0.001s
I20250831 07:59:34.479629 32464 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
W20250831 07:59:34.481053 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:34.482033 31699 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:34.487321 1275 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:34.483968 1740 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:34.485761 1804 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:34.472111 1378 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:34.489019 1804 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:34.490553 1995 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 10 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:34.491035 1804 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:34.491457 2117 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.705s user 0.032s sys 0.025s
I20250831 07:59:34.509814 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:34.515389 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.491925 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:34.516283 31808 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
W20250831 07:59:34.532171 32436 leader_election.cc:336] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.685s (SENT)
I20250831 07:59:34.534530 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:34.504415 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.535656 31804 raft_consensus.cc:1238] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 7. Current term is 8. Ops: []
W20250831 07:59:34.496345 2183 log.cc:927] Time spent T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20250831 07:59:34.540380 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:34.500264 2174 log.cc:927] Time spent T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.257s user 0.000s sys 0.001s
W20250831 07:59:34.491811 2117 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.719s user 0.032s sys 0.025s
I20250831 07:59:34.504887 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:34.547338 31805 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
W20250831 07:59:34.510102 2173 log.cc:927] Time spent T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.225s user 0.000s sys 0.001s
I20250831 07:59:34.540918 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:34.547636 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 66000.6us
I20250831 07:59:34.494772 1897 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:34.563630 1275 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:34.565912 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:34.529675 1740 raft_consensus.cc:695] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 6 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:34.571152 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.500346 1867 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:34.571887 31726 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
W20250831 07:59:34.574783 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:34.579922 31732 raft_consensus.cc:1238] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:34.583572 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:34.592449 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:34.594050 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:34.596316 31806 raft_consensus.cc:1238] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20250831 07:59:34.596406 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:34.597091 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1411) took 1195 ms. Trace:
I20250831 07:59:34.600525 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:34.604882 31731 raft_consensus.cc:1238] T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c [term 3 LEADER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 2. Current term is 3. Ops: []
I20250831 07:59:34.606467 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:34.607038 31729 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:34.607647 31734 raft_consensus.cc:3058] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 6
I20250831 07:59:34.609684 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:34.610626 31809 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:34.612950 31800 raft_consensus.cc:1273] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250831 07:59:34.613032 31730 raft_consensus.cc:1215] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Deduplicated request from leader. Original: 4.6->[6.7-6.7] Dedup: 6.7->[]
I20250831 07:59:34.614554 31730 raft_consensus.cc:1238] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250831 07:59:34.597560 31733 rpcz_store.cc:276] 0831 07:59:33.402006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:33.412427 (+ 10421us) service_pool.cc:225] Handling call
0831 07:59:33.436408 (+ 23981us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:33.444030 (+ 7622us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:33.444078 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:33.461646 (+ 17568us) log.cc:844] Serialized 26 byte log entry
0831 07:59:33.906552 (+444906us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:33.906617 (+ 65us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:33.906724 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:34.584407 (+677683us) raft_consensus.cc:1683] finished
0831 07:59:34.584431 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:34.597033 (+ 12602us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11228,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":6626,"thread_start_us":346046,"threads_started":1,"wal-append.queue_time_us":888433}
I20250831 07:59:34.613538 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.616303 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:34.617106 31731 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
I20250831 07:59:34.617381 31880 raft_consensus.cc:2408] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250831 07:59:34.614312 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.618467 31732 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:34.582085 2124 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:34.620700 1275 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:34.622032 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.613641 31800 pending_rounds.cc:85] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 3
I20250831 07:59:34.580714 1897 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:34.592308 2127 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:34.625615 1897 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:34.625684 2127 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:34.625188 31800 pending_rounds.cc:107] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250831 07:59:34.627700 1897 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:34.627951 31800 raft_consensus.cc:2887] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:34.629329 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:34.632194 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.633152 31803 raft_consensus.cc:2408] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20250831 07:59:34.635879 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:34.636746 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.638938 1275 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:34.640836 31800 raft_consensus.cc:1273] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20250831 07:59:34.616488 2190 log.cc:927] Time spent T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.132s user 0.001s sys 0.001s
I20250831 07:59:34.649377 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.635926 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:34.652925 31730 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 6
I20250831 07:59:34.654682 31805 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 8.
I20250831 07:59:34.654726 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.656358 1275 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:34.656663 31808 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:34.657904 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:34.661736 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.663980 31734 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:34.665055 31727 raft_consensus.cc:1273] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:34.665783 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.637691 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:34.666723 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 60368.7us
I20250831 07:59:34.668180 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.669189 31805 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:34.670408 31802 raft_consensus.cc:2366] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:34.627725 2127 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:34.670293 31807 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:34.673620 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:34.613874 2100 consensus_queue.cc:1046] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.052s
I20250831 07:59:34.682910 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:34.681342 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.688289 31733 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
W20250831 07:59:34.690536 2181 log.cc:927] Time spent T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
W20250831 07:59:34.693186 2179 log.cc:927] Time spent T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.408s user 0.003s sys 0.000s
I20250831 07:59:34.649709 1740 consensus_queue.cc:237] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:34.653318 2124 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:34.696141 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.697844 2124 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:34.699412 2100 consensus_queue.cc:1230] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer a0d845d01c8c4894895f66e3beae298c log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 07:59:34.701145 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:34.704252 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:34.671420 2087 consensus_queue.cc:1035] T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20250831 07:59:34.706248 1867 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:34.707871 1843 raft_consensus.cc:3053] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:34.708385 1843 raft_consensus.cc:738] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:34.709445 1843 consensus_queue.cc:260] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:34.711028 1843 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:34.711589 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:34.715945 2178 log.cc:927] Time spent T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.431s user 0.000s sys 0.009s
I20250831 07:59:34.718115 32464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38366 (request call id 492) took 1406 ms. Trace:
I20250831 07:59:34.718670 32464 rpcz_store.cc:276] 0831 07:59:33.311176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:33.311652 (+ 476us) service_pool.cc:225] Handling call
0831 07:59:34.717835 (+1406183us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b5400500fd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:34.718006 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":822027008}
I20250831 07:59:34.720623 31729 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:34.744249 31731 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 07:59:34.749647 31730 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:34.756525 31732 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 07:59:34.757493 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.758267 31730 raft_consensus.cc:2374] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6. Re-sending same reply.
I20250831 07:59:34.759444 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:34.759544 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 154932us
I20250831 07:59:34.760174 31807 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:34.760206 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.761428 31803 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:34.767853 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:34.768111 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.768685 31730 raft_consensus.cc:2374] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8. Re-sending same reply.
I20250831 07:59:34.768936 31808 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:34.770159 31802 raft_consensus.cc:1238] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20250831 07:59:34.756723 2135 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:34.783043 31734 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
W20250831 07:59:34.772567 31849 leader_election.cc:336] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.920s (SENT)
W20250831 07:59:34.786770 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 180718us
I20250831 07:59:34.789714 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.790145 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:34.790319 31734 raft_consensus.cc:2374] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7. Re-sending same reply.
I20250831 07:59:34.790916 31730 raft_consensus.cc:2374] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6. Re-sending same reply.
I20250831 07:59:34.792032 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:34.792717 31730 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:34.793071 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:34.794166 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:34.794924 31729 raft_consensus.cc:2374] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6. Re-sending same reply.
I20250831 07:59:34.755935 1966 consensus_queue.cc:1035] T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20250831 07:59:34.766043 2166 consensus_queue.cc:1035] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20250831 07:59:34.797546 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.800958 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.796775 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:34.799237 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.799293 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:34.800176 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.800287 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.803666 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.810581 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.812109 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.812990 31807 raft_consensus.cc:1238] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:34.813982 31730 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:34.814847 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.816602 31728 raft_consensus.cc:2391] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:34.817998 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.649561 2096 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:34.824074 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.821627 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:34.822563 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.826308 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:34.827232 2192 log.cc:927] Time spent T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
I20250831 07:59:34.830261 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:34.825726 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:34.837803 2099 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:34.838356 2099 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:34.840845 2168 log.cc:927] Time spent T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.555s user 0.005s sys 0.009s
I20250831 07:59:34.842115 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1793) took 1368 ms. Trace:
I20250831 07:59:34.842448 31874 rpcz_store.cc:276] 0831 07:59:33.473553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:33.473697 (+ 144us) service_pool.cc:225] Handling call
0831 07:59:33.550826 (+ 77129us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:33.570212 (+ 19386us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:33.570241 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:33.608165 (+ 37924us) log.cc:844] Serialized 26 byte log entry
0831 07:59:33.912533 (+304368us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:33.940772 (+ 28239us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:33.940869 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:34.840274 (+899405us) spinlock_profiling.cc:243] Waited 5.48 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:34.841831 (+ 1557us) raft_consensus.cc:1683] finished
0831 07:59:34.841850 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:34.842041 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10736,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":37888,"spinlock_wait_cycles":12613376,"thread_start_us":268401,"threads_started":1,"wal-append.queue_time_us":576175}
I20250831 07:59:34.845671 32464 raft_consensus.cc:1238] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: []
W20250831 07:59:34.846098 31773 leader_election.cc:336] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 2.000s (SENT)
W20250831 07:59:34.847313 31773 leader_election.cc:336] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 3.166s (SENT)
I20250831 07:59:34.847772 31773 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:34.847997 2204 consensus_queue.cc:1046] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250831 07:59:34.848884 2204 raft_consensus.cc:3053] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Stepping down as leader of term 5
W20250831 07:59:34.849115 31773 leader_election.cc:336] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 3.272s (SENT)
I20250831 07:59:34.849972 1804 raft_consensus.cc:2747] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:34.796049 2135 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:34.849259 2204 raft_consensus.cc:738] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
W20250831 07:59:34.853293 2184 log.cc:927] Time spent T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.568s user 0.003s sys 0.000s
I20250831 07:59:34.852116 2204 consensus_queue.cc:260] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:34.859650 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.860350 31804 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:34.862733 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:34.795754 2186 consensus_queue.cc:1035] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20250831 07:59:34.867435 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:34.844512 2185 log.cc:927] Time spent T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.179s user 0.000s sys 0.000s
I20250831 07:59:34.868137 31804 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:34.869442 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.864518 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.870270 31730 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:34.849829 31773 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:34.871759 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.862787 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.876521 31804 raft_consensus.cc:1238] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:34.877135 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.863870 31726 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Advancing to term 9
W20250831 07:59:34.845804 2172 log.cc:927] Time spent T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.615s user 0.001s sys 0.002s
I20250831 07:59:34.870986 2204 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:34.885913 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.864342 1804 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:34.897924 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 133560us
I20250831 07:59:34.898715 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:34.907552 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:34.884339 2087 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:34.923763 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:34.900085 31775 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:34.920295 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.927434 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:34.928352 31805 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
W20250831 07:59:34.930835 31849 leader_election.cc:336] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.589s (SENT)
W20250831 07:59:34.940457 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 91363.3us
I20250831 07:59:34.942826 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:34.943869 31725 raft_consensus.cc:2366] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6.
I20250831 07:59:34.945832 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 264) took 1289 ms. Trace:
I20250831 07:59:34.950451 31725 raft_consensus.cc:1238] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:34.946367 31878 rpcz_store.cc:276] 0831 07:59:33.656559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:33.734726 (+ 78167us) service_pool.cc:225] Handling call
0831 07:59:33.750392 (+ 15666us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:33.760791 (+ 10399us) raft_consensus.cc:1532] Early marking committed up to index 7
0831 07:59:33.800527 (+ 39736us) log.cc:844] Serialized 14 byte log entry
0831 07:59:34.714257 (+913730us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b18001f9360. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce501365a 00007fdcee4e4409 00007fdcebebce6f 00007fdce509b90e 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d
0831 07:59:34.840215 (+125958us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b18001f9360. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce5069786 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce5069786 00007fdce5069720
0831 07:59:34.945406 (+105191us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:34.945476 (+ 70us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:34.945543 (+ 67us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:34.945630 (+ 87us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:34.945767 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6990,"prepare.queue_time_us":63091,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":2189,"raft.queue_time_us":6785,"spinlock_wait_cycles":85783936,"thread_start_us":90403,"threads_started":1}
I20250831 07:59:34.901700 2166 consensus_queue.cc:1035] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.043s
W20250831 07:59:34.958009 32434 leader_election.cc:336] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 3.156s (SENT)
I20250831 07:59:34.938856 2146 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:34.898820 1804 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:34.948763 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 48787us
I20250831 07:59:34.963263 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.953003 2116 raft_consensus.cc:2802] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:34.969456 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.930398 2087 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:34.984510 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:34.961305 2166 consensus_queue.cc:1230] T 6b2e336f43934f118ecacdc1e1d2da60 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250831 07:59:34.961094 2135 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:34.987376 31725 raft_consensus.cc:1273] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 07:59:34.988040 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:34.985699 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:34.978404 2176 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.015s sys 0.021s
I20250831 07:59:35.039434 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:34.971709 2116 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:35.042234 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:34.965410 2117 consensus_queue.cc:1046] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20250831 07:59:35.062309 31725 raft_consensus.cc:1273] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:35.095201 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:35.112727 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1786) took 1799 ms. Trace:
I20250831 07:59:35.114683 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:35.115324 31874 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
I20250831 07:59:35.116474 31805 raft_consensus.cc:1238] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:35.117189 31848 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:35.039883 2176 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.754s user 0.015s sys 0.021s
I20250831 07:59:34.966105 2205 consensus_queue.cc:1035] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.088s
I20250831 07:59:35.120462 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:35.143127 31725 raft_consensus.cc:2374] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 8. Re-sending same reply.
I20250831 07:59:35.113122 31881 rpcz_store.cc:276] 0831 07:59:33.312914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:33.313049 (+ 135us) service_pool.cc:225] Handling call
0831 07:59:33.313431 (+ 382us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:33.387694 (+ 74263us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 07:59:33.809599 (+421905us) log.cc:844] Serialized 14 byte log entry
0831 07:59:34.714701 (+905102us) log.cc:844] Serialized 14 byte log entry
0831 07:59:34.714907 (+ 206us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:34.714955 (+ 48us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:34.715001 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:35.112521 (+397520us) spinlock_profiling.cc:243] Waited 74 ms on lock 0x7b540007cbd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:35.112565 (+ 44us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:35.112674 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":734031,"prepare.queue_time_us":1051222,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":69,"raft.queue_time_us":2293,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":197,"spinlock_wait_cycles":170280320,"thread_start_us":543298,"threads_started":2,"wal-append.queue_time_us":419864}
I20250831 07:59:35.124208 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:35.125531 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:35.056111 2116 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:35.149273 31727 raft_consensus.cc:1273] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:35.150153 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:35.152675 31877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 278) took 1332 ms. Trace:
I20250831 07:59:35.152927 31877 rpcz_store.cc:276] 0831 07:59:33.840437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:35.149965 (+1309528us) service_pool.cc:225] Handling call
0831 07:59:35.152630 (+ 2665us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:35.110862 2132 consensus_queue.cc:1035] T 518e6f5a91554de78e0bd0b6ec04c12a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.034s
I20250831 07:59:35.165627 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:35.162140 32436 leader_election.cc:336] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.823s (SENT)
W20250831 07:59:35.156222 2173 log.cc:927] Time spent T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20250831 07:59:35.184536 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:35.150843 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:35.180320 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 185306us
W20250831 07:59:35.192804 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:35.193352 31773 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:35.196377 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:35.197068 31729 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:35.160300 2096 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:35.195655 31849 leader_election.cc:336] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.183s (SENT)
I20250831 07:59:35.198289 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:35.201399 31849 leader_election.cc:336] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.662s (SENT)
W20250831 07:59:35.202726 31849 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:35.206841 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:35.180025 2201 raft_consensus.cc:2747] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:35.260860 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:35.276437 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:35.177332 2218 consensus_queue.cc:1046] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.084s
W20250831 07:59:35.289171 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:35.290853 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:35.276352 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:35.294226 31879 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:35.294690 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 276) took 1475 ms. Trace:
I20250831 07:59:35.295482 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:35.264495 2174 log.cc:927] Time spent T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250831 07:59:35.298683 31728 raft_consensus.cc:2391] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:35.298585 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:35.294932 31879 rpcz_store.cc:276] 0831 07:59:33.819331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:35.178481 (+1359150us) service_pool.cc:225] Handling call
0831 07:59:35.294646 (+116165us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:35.300681 2210 log.cc:927] Time spent T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.313s user 0.001s sys 0.000s
W20250831 07:59:35.301642 2206 log.cc:927] Time spent T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.329s user 0.001s sys 0.000s
W20250831 07:59:35.302788 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 131333us
W20250831 07:59:35.304423 2214 log.cc:927] Time spent T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.258s user 0.000s sys 0.000s
W20250831 07:59:35.305676 2161 log.cc:927] Time spent T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 1.339s user 0.009s sys 0.022s
I20250831 07:59:35.284389 2125 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 07:59:35.310885 2212 log.cc:927] Time spent T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20250831 07:59:35.312345 1804 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:35.313539 2087 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:35.319231 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:35.320405 2218 raft_consensus.cc:3053] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:35.320736 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:35.320837 2218 raft_consensus.cc:738] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:35.321202 1947 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:35.321538 31729 raft_consensus.cc:2366] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6.
W20250831 07:59:35.321995 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:35.321817 2218 consensus_queue.cc:260] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:35.322289 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:35.323041 2218 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:35.323343 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:35.320399 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:35.324421 31734 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:35.324599 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 406) took 1703 ms. Trace:
I20250831 07:59:35.325237 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:35.326608 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:35.328488 31726 raft_consensus.cc:2466] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 9.
I20250831 07:59:35.331359 31726 raft_consensus.cc:1238] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 10 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 9. Current term is 10. Ops: []
I20250831 07:59:35.332924 32436 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.324976 32469 rpcz_store.cc:276] 0831 07:59:33.621395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:33.621598 (+ 203us) service_pool.cc:225] Handling call
0831 07:59:33.632582 (+ 10984us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:33.632784 (+ 202us) raft_consensus.cc:1532] Early marking committed up to index 8
0831 07:59:33.869036 (+236252us) log.cc:844] Serialized 14 byte log entry
0831 07:59:35.324166 (+1455130us) spinlock_profiling.cc:243] Waited 8.14 ms on lock 0x7b18003ae7c0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce7c4955c 00007fdce507e77b 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce5098805 00007fdce7ff5020 00007fdce80076c6 00007fdce8018237
0831 07:59:35.324268 (+ 102us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:35.324313 (+ 45us) raft_consensus.cc:1643] Marking committed up to 8
0831 07:59:35.324363 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:35.324424 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:35.324525 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6339,"prepare.queue_time_us":792160,"prepare.run_cpu_time_us":4230,"prepare.run_wall_time_us":23195,"raft.queue_time_us":73837,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":16429,"spinlock_wait_cycles":18732288,"thread_start_us":653147,"threads_started":3,"wal-append.queue_time_us":445464}
I20250831 07:59:35.219770 2096 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:35.336495 32436 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.337515 32436 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:35.341032 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:35.339720 2092 raft_consensus.cc:2747] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 07:59:35.346009 2131 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250831 07:59:35.346343 2223 consensus_queue.cc:1046] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:35.339594 2205 raft_consensus.cc:2747] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:35.342810 2166 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:35.348232 2166 raft_consensus.cc:695] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:35.349099 2166 consensus_queue.cc:237] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:35.350252 2195 log.cc:927] Time spent T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.660s user 0.003s sys 0.012s
I20250831 07:59:35.347194 2201 raft_consensus.cc:3053] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:35.370631 31699 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:35.373335 31803 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
W20250831 07:59:35.384299 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 267184us
I20250831 07:59:35.385133 31848 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.340620 2125 consensus_queue.cc:1046] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
W20250831 07:59:35.391431 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 137367us
I20250831 07:59:35.391605 31849 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:35.394677 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 44838.8us
I20250831 07:59:35.369637 2201 raft_consensus.cc:738] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
W20250831 07:59:35.392473 2182 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.691s user 0.010s sys 0.010s
I20250831 07:59:35.399957 31848 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.369853 1378 consensus_queue.cc:1035] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.025s
I20250831 07:59:35.400710 31849 leader_election.cc:400] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Vote denied by peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027) with higher term. Message: Invalid argument: T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6.
I20250831 07:59:35.372184 2157 consensus_queue.cc:1035] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20250831 07:59:35.370710 2205 consensus_queue.cc:1035] T 000f9ab653554f498fe14065e6656a62 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20250831 07:59:35.433688 31849 leader_election.cc:403] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250831 07:59:35.434000 31728 raft_consensus.cc:1238] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 6 LEADER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: []
W20250831 07:59:35.398926 2182 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s user 0.010s sys 0.010s
I20250831 07:59:35.399806 2204 raft_consensus.cc:2747] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:35.442585 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 45180.5us
W20250831 07:59:35.442072 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 44330.7us
I20250831 07:59:35.444314 31848 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.400125 2230 raft_consensus.cc:2747] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 07:59:35.436931 2137 raft_consensus.cc:2747] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:35.445997 31699 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:35.449396 31848 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.447685 2204 raft_consensus.cc:2747] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:35.453624 2231 consensus_queue.cc:1035] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:59:35.441461 2224 raft_consensus.cc:2747] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 07:59:35.456399 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 44711.4us
I20250831 07:59:35.459544 31848 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.435742 1962 consensus_queue.cc:1035] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20250831 07:59:35.410836 2227 consensus_queue.cc:1046] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20250831 07:59:35.468081 31699 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:35.466243 31848 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.456164 2208 raft_consensus.cc:2747] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027) with higher term. Message: Invalid argument: T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6.
I20250831 07:59:35.458628 2231 consensus_queue.cc:1230] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 7512ba2ed3a94982a22a75f51a647e65 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 07:59:35.460664 2125 consensus_queue.cc:1046] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20250831 07:59:35.459540 2230 raft_consensus.cc:2747] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:35.475428 31699 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:35.475612 31848 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:35.476356 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 65327.3us
I20250831 07:59:35.477010 31848 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:35.483016 2099 consensus_meta.cc:287] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.644s user 0.004s sys 0.004s
I20250831 07:59:35.483692 2220 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:35.483888 2116 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:35.467214 2223 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:35.474076 2137 raft_consensus.cc:2747] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:35.497866 2223 raft_consensus.cc:695] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:35.498848 2223 consensus_queue.cc:237] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:35.499248 2146 raft_consensus.cc:3053] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:35.476891 2204 raft_consensus.cc:2747] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:35.446751 2201 consensus_queue.cc:260] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:35.490403 2208 raft_consensus.cc:2747] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:35.502171 31775 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:35.512766 2244 consensus_queue.cc:1035] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20250831 07:59:35.516340 1947 consensus_queue.cc:1035] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:35.516754 2137 raft_consensus.cc:2747] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:35.494313 2099 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:35.535013 2237 log.cc:927] Time spent T 9d510d241da34d25a50f2d65d17041a0 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20250831 07:59:35.517146 2226 consensus_queue.cc:1046] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:59:35.517175 2228 raft_consensus.cc:3053] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:35.518137 2157 consensus_queue.cc:1035] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:35.518518 1804 consensus_queue.cc:1046] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:59:35.519126 2227 raft_consensus.cc:2802] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 07:59:35.512272 2116 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:35.514230 2244 raft_consensus.cc:2802] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:35.544705 2244 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:35.539840 2228 raft_consensus.cc:738] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:35.545147 2244 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:35.545704 2228 consensus_queue.cc:260] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:35.522761 31848 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.504783 2146 raft_consensus.cc:738] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:35.537815 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:35.538259 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:35.510039 2117 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:35.542968 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:35.556392 31801 raft_consensus.cc:2391] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:35.556306 2117 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:35.557348 2227 raft_consensus.cc:695] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:35.558451 2239 log.cc:927] Time spent T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.106s user 0.002s sys 0.000s
I20250831 07:59:35.558297 2227 consensus_queue.cc:237] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:35.559954 2211 log.cc:927] Time spent T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.521s user 0.002s sys 0.000s
I20250831 07:59:35.550335 31809 raft_consensus.cc:2391] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:35.549520 2220 raft_consensus.cc:2802] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:35.558379 2117 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:35.551535 31848 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.569187 31848 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:35.571914 31848 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.537223 2099 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:35.612795 31848 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:35.623112 2204 consensus_queue.cc:1046] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:59:35.624401 2204 raft_consensus.cc:3053] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:35.624753 2204 raft_consensus.cc:738] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:35.625615 2204 consensus_queue.cc:260] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:35.626971 2204 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:35.573472 1378 raft_consensus.cc:2747] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:35.574095 2220 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:35.630506 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 155146us
I20250831 07:59:35.632390 31882 raft_consensus.cc:1273] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:35.617237 2208 raft_consensus.cc:2747] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:35.636410 31848 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.634435 31699 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:35.635082 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 117493us
I20250831 07:59:35.639199 31848 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:35.633435 2220 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:35.641567 2099 consensus_queue.cc:1035] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:59:35.635696 2137 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:35.646126 31848 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:35.644840 31773 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.647078 2246 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:35.652356 31699 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:35.652613 31848 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:35.651607 1897 raft_consensus.cc:2802] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:35.638371 32463 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38366 (request call id 496) took 2267 ms. Trace:
I20250831 07:59:35.639494 1378 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:35.636368 2249 raft_consensus.cc:2747] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250831 07:59:35.658409 31773 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.649380 2208 raft_consensus.cc:2747] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 07:59:35.662950 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:35.655200 1897 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:35.663638 31733 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:35.669287 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:35.673458 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:35.674419 31801 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250831 07:59:35.676139 31773 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.677779 31773 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.657375 1378 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:35.667155 1897 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:35.684509 2242 raft_consensus.cc:2747] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:35.685556 2247 consensus_queue.cc:1035] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20250831 07:59:35.688042 32464 raft_consensus.cc:1273] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:35.688104 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:35.688596 32470 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250831 07:59:35.688868 31725 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:35.656509 2137 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:35.639170 2146 consensus_queue.cc:260] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:35.699944 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:35.664192 2117 raft_consensus.cc:2747] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:35.703425 31725 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:35.664355 2246 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:35.678978 2157 raft_consensus.cc:2747] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 07:59:35.714334 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 62312.4us
I20250831 07:59:35.714970 2132 consensus_queue.cc:1046] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250831 07:59:35.678601 1995 raft_consensus.cc:2802] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:35.719205 31775 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:35.721079 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 68808us
W20250831 07:59:35.723625 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 45956.5us
I20250831 07:59:35.702054 1378 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:35.725826 31773 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.729596 31773 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.732625 31699 leader_election.cc:400] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:35.733732 31699 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:35.661074 2245 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20250831 07:59:35.735283 31699 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:35.692534 1897 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:35.741106 31699 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:35.722957 1995 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:35.725761 2218 raft_consensus.cc:2747] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:35.744599 31848 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:35.734954 2099 raft_consensus.cc:2747] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:35.732613 2117 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election won for term 7
W20250831 07:59:35.738015 2142 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.370s user 0.020s sys 0.022s
I20250831 07:59:35.746857 2224 consensus_queue.cc:1046] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20250831 07:59:35.758735 2224 raft_consensus.cc:3053] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:35.759078 2224 raft_consensus.cc:738] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:35.759302 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 273) took 1112 ms. Trace:
I20250831 07:59:35.752295 2258 raft_consensus.cc:2747] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 07:59:35.761088 2117 raft_consensus.cc:695] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:35.761385 2196 log.cc:927] Time spent T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 1.080s user 0.005s sys 0.008s
I20250831 07:59:35.754424 2250 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:35.755149 1897 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:35.759663 31732 rpcz_store.cc:276] 0831 07:59:34.647200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:34.651446 (+ 4246us) service_pool.cc:225] Handling call
0831 07:59:34.753268 (+101822us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:34.756923 (+ 3655us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:35.759113 (+1002190us) spinlock_profiling.cc:243] Waited 84 ms on lock 0x7b54000104d0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:35.759247 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":193265536}
I20250831 07:59:35.770092 31773 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.743374 2225 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:35.741677 2182 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.013s
I20250831 07:59:35.749514 1995 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:35.778936 31699 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:35.782385 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:35.748128 2217 raft_consensus.cc:2747] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 07:59:35.764575 2142 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.392s user 0.020s sys 0.025s
I20250831 07:59:35.786697 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:35.757717 2246 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:35.789469 32434 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.790769 2258 raft_consensus.cc:2802] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:35.791162 2258 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:35.657809 32463 rpcz_store.cc:276] 0831 07:59:33.366027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:33.400659 (+ 34632us) service_pool.cc:225] Handling call
0831 07:59:33.462960 (+ 62301us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:33.539446 (+ 76486us) raft_consensus.cc:1532] Early marking committed up to index 7
0831 07:59:33.955224 (+415778us) log.cc:844] Serialized 14 byte log entry
0831 07:59:34.300056 (+344832us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:34.300695 (+ 639us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:34.301228 (+ 533us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:34.834026 (+532798us) spinlock_profiling.cc:243] Waited 5.04 ms on lock 0x7b18004d4d00. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b792 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:59:34.837016 (+ 2990us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b18004d4d00. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce5069786 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89
0831 07:59:35.629130 (+792114us) spinlock_profiling.cc:243] Waited 76.2 ms on lock 0x7b540062f054. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:35.629918 (+ 788us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:35.633650 (+ 3732us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":666119,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":82999,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":10991,"spinlock_wait_cycles":195397120,"thread_start_us":638119,"threads_started":2,"wal-append.queue_time_us":611561}
I20250831 07:59:35.791512 2258 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:35.792055 2256 raft_consensus.cc:2802] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:35.792546 31773 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:35.770056 2250 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:35.799422 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:35.802255 2242 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:35.798380 31733 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:35.806421 31775 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:35.818029 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:35.819187 31800 raft_consensus.cc:2391] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:35.823691 2242 raft_consensus.cc:2747] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:35.824055 2246 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:35.825027 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:35.832850 31848 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:35.756433 2251 consensus_queue.cc:1035] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20250831 07:59:35.762038 2117 consensus_queue.cc:237] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:35.828984 31773 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.829746 31879 raft_consensus.cc:1273] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250831 07:59:35.831774 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 94062.8us
I20250831 07:59:35.828578 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:35.793561 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38366 (request call id 505) took 1364 ms. Trace:
I20250831 07:59:35.784544 2226 raft_consensus.cc:2747] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:35.842904 31878 raft_consensus.cc:2391] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:35.835510 31881 pending_rounds.cc:85] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10: Aborting all ops after (but not including) 6
I20250831 07:59:35.844237 31698 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:35.858067 31881 pending_rounds.cc:107] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20250831 07:59:35.859222 31881 raft_consensus.cc:2887] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:35.845142 31773 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.836308 2099 raft_consensus.cc:2802] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 07:59:35.800232 2256 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:35.868333 31773 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.827409 2250 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:35.833331 2186 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:35.759943 2224 consensus_queue.cc:260] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:35.842252 2217 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:35.839720 2116 raft_consensus.cc:2802] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:35.860764 2242 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
W20250831 07:59:35.904785 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 49229.5us
I20250831 07:59:35.891548 2186 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:35.905735 2186 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:35.864527 2231 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 07:59:35.838706 2157 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.051s
I20250831 07:59:35.892302 2256 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:35.873713 2227 raft_consensus.cc:2747] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 07:59:35.917080 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 121678us
I20250831 07:59:35.874877 2099 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:35.917577 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 78082.4us
I20250831 07:59:35.785404 2225 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:35.931622 31699 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:35.913271 2116 raft_consensus.cc:695] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:35.918164 2137 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:35.956841 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:35.957602 31875 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:35.934818 2099 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:35.931583 2242 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:35.930820 2224 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:35.912827 2226 consensus_queue.cc:1035] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20250831 07:59:35.839181 2246 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:35.844322 32466 rpcz_store.cc:276] 0831 07:59:34.424694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:34.429080 (+ 4386us) service_pool.cc:225] Handling call
0831 07:59:34.477600 (+ 48520us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:34.510920 (+ 33320us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:34.510975 (+ 55us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:34.511645 (+ 670us) log.cc:844] Serialized 26 byte log entry
0831 07:59:34.568852 (+ 57207us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:34.570100 (+ 1248us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:34.580892 (+ 10792us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:35.760700 (+1179808us) spinlock_profiling.cc:243] Waited 3.56 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:35.762449 (+ 1749us) raft_consensus.cc:1683] finished
0831 07:59:35.769250 (+ 6801us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:35.787609 (+ 18359us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32836,"raft.queue_time_us":34317,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":9646,"spinlock_wait_cycles":8179968,"thread_start_us":44351,"threads_started":1,"wal-append.queue_time_us":102332}
I20250831 07:59:35.953614 2227 raft_consensus.cc:2747] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:35.955945 2251 raft_consensus.cc:2747] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250831 07:59:35.971112 2116 consensus_queue.cc:237] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:35.903735 2217 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:35.991039 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 122821us
I20250831 07:59:35.993029 31698 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:35.993117 31773 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.994849 31773 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:35.998649 31773 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:36.010835 31849 leader_election.cc:336] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.977s (SENT)
I20250831 07:59:36.011566 31849 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:36.017225 31699 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:36.023548 2271 raft_consensus.cc:2747] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:36.029417 2137 raft_consensus.cc:2747] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:36.032471 2278 raft_consensus.cc:2747] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:36.033229 2278 raft_consensus.cc:2747] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:36.033869 2278 raft_consensus.cc:2747] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 07:59:36.020438 2265 log.cc:927] Time spent T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20250831 07:59:35.975437 2242 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:36.043325 2137 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:36.045934 2271 consensus_queue.cc:1035] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250831 07:59:36.048590 2268 log.cc:927] Time spent T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20250831 07:59:36.049072 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1710) took 1256 ms. Trace:
I20250831 07:59:36.035827 2201 raft_consensus.cc:3058] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:36.052027 2216 raft_consensus.cc:3053] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:36.053154 2217 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:36.054468 2213 log.cc:927] Time spent T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.758s user 0.001s sys 0.001s
I20250831 07:59:36.059070 2231 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:36.060290 2257 log.cc:927] Time spent T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.220s user 0.001s sys 0.000s
I20250831 07:59:36.061825 2236 raft_consensus.cc:3053] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:36.024974 2269 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:36.063669 2104 raft_consensus.cc:3053] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:35.996182 2148 consensus_queue.cc:1035] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.200s
I20250831 07:59:36.064914 2225 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:36.034744 2230 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:36.076659 2216 raft_consensus.cc:738] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:36.067134 2246 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:36.049783 31808 raft_consensus.cc:1273] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 07:59:36.075299 31803 rpcz_store.cc:276] 0831 07:59:34.792704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:34.792841 (+ 137us) service_pool.cc:225] Handling call
0831 07:59:36.048885 (+1256044us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b54000ce9d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:36.048995 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":917880704}
W20250831 07:59:36.098702 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 91557.6us
I20250831 07:59:36.107007 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:36.088192 2236 raft_consensus.cc:738] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:36.109046 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:36.075737 2137 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:36.110350 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 93279.3us
I20250831 07:59:36.111238 31699 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:36.109907 31877 raft_consensus.cc:2391] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:36.090530 2104 raft_consensus.cc:738] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:36.091692 2269 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:36.113941 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:36.117514 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:36.119166 31801 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:36.095901 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:36.128722 31878 raft_consensus.cc:1238] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250831 07:59:36.130043 31699 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:36.117548 2251 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:36.117625 2244 consensus_meta.cc:287] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.572s user 0.012s sys 0.000s
I20250831 07:59:36.135178 2137 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:36.135022 2244 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.136487 2242 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:36.142750 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1872) took 1271 ms. Trace:
I20250831 07:59:36.143170 31730 rpcz_store.cc:276] 0831 07:59:34.871663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:34.873256 (+ 1593us) service_pool.cc:225] Handling call
0831 07:59:34.873569 (+ 313us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:34.873807 (+ 238us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:34.873832 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:34.874274 (+ 442us) log.cc:844] Serialized 26 byte log entry
0831 07:59:34.989841 (+115567us) raft_consensus.cc:1643] Marking committed up to 2
0831 07:59:34.989874 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:34.989943 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:36.053815 (+1063872us) spinlock_profiling.cc:243] Waited 8.79 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:36.128391 (+ 74576us) raft_consensus.cc:1683] finished
0831 07:59:36.128413 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:36.142684 (+ 14271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20216704,"thread_start_us":115184,"threads_started":1,"wal-append.queue_time_us":288673}
I20250831 07:59:36.148820 31730 raft_consensus.cc:1238] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:36.149307 31728 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250831 07:59:36.096019 2231 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.155186 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:36.155144 2236 consensus_queue.cc:260] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.155840 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:36.157526 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:36.159552 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:36.160869 2104 consensus_queue.cc:260] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.161942 2104 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:36.162374 1378 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.163666 31876 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8. Ops: []
I20250831 07:59:36.167935 1995 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.168979 31732 raft_consensus.cc:1238] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20250831 07:59:36.169202 2137 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:36.169979 1995 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:36.170513 1947 raft_consensus.cc:3053] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:36.170905 1947 raft_consensus.cc:738] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:36.171557 2186 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:36.171655 1378 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:36.166437 31728 raft_consensus.cc:1273] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250831 07:59:36.174144 2186 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:36.174413 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:36.174975 31730 raft_consensus.cc:2374] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7. Re-sending same reply.
W20250831 07:59:36.175918 2220 consensus_meta.cc:287] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.520s user 0.008s sys 0.003s
I20250831 07:59:36.176345 2220 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.156447 2231 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:36.181672 31849 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:36.181521 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:36.181697 31882 raft_consensus.cc:2391] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:36.120299 2269 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:36.186328 31727 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 07:59:36.186609 31849 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:36.188660 31849 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:36.192039 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:36.194852 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:36.183992 2087 raft_consensus.cc:2747] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:36.196624 31727 raft_consensus.cc:1273] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 07:59:36.196915 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:36.197193 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:36.198117 31849 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:36.171751 1947 consensus_queue.cc:260] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.206842 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:36.208381 31726 raft_consensus.cc:2391] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:36.208530 31849 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:36.209924 31773 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:36.219707 31725 raft_consensus.cc:1238] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
W20250831 07:59:36.214082 2204 consensus_meta.cc:287] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.587s user 0.011s sys 0.003s
I20250831 07:59:36.193301 2230 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250831 07:59:36.281841 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 54225.6us
W20250831 07:59:36.284174 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 70081.9us
I20250831 07:59:36.205819 2096 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:36.289299 31878 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:36.297911 31849 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:36.299518 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:36.299958 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:36.302111 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:36.302516 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:36.302877 31800 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8.
I20250831 07:59:36.304210 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:36.305359 31809 pending_rounds.cc:85] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Aborting all ops after (but not including) 2
I20250831 07:59:36.305727 31809 pending_rounds.cc:107] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250831 07:59:36.306020 31809 raft_consensus.cc:2887] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:36.309469 31849 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:36.202610 2246 consensus_queue.cc:1046] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.076s
I20250831 07:59:36.263859 1378 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:36.202225 2225 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.091s
I20250831 07:59:36.264060 2220 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:36.212059 2166 consensus_queue.cc:1035] T d3d650c422ee41af95705f3fd876ddb8 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.075s
I20250831 07:59:36.322638 31699 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:36.202951 2205 consensus_queue.cc:1046] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.094s
I20250831 07:59:36.178584 2216 consensus_queue.cc:260] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.323652 1995 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:36.358604 1995 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:36.360730 1995 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:36.337354 2230 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:36.343199 2092 raft_consensus.cc:3053] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Stepping down as leader of term 8
I20250831 07:59:36.348348 2280 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
W20250831 07:59:36.350353 2194 log.cc:927] Time spent T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 1.684s user 0.003s sys 0.018s
W20250831 07:59:36.377928 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 55386.5us
W20250831 07:59:36.378379 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 166568us
I20250831 07:59:36.393193 31773 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:36.393698 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:36.424974 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:36.336268 2157 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:36.305549 2116 consensus_queue.cc:1046] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20250831 07:59:36.437744 31800 raft_consensus.cc:1238] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 LEADER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20250831 07:59:36.320698 1897 consensus_queue.cc:1035] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20250831 07:59:36.377316 2125 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:36.336642 2224 consensus_queue.cc:1046] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
I20250831 07:59:36.377192 2092 raft_consensus.cc:738] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:36.458148 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:36.423019 2231 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 07:59:36.428821 2236 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:36.324532 2258 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.500249 31726 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250831 07:59:36.452855 2262 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:36.370273 2256 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.515755 31732 raft_consensus.cc:1238] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:36.518564 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:36.479929 2231 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:36.530267 31729 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:36.468639 2125 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:36.500389 2264 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:36.504634 2244 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:36.452522 2157 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.518573 2259 raft_consensus.cc:2747] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:36.545894 31800 raft_consensus.cc:1273] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 07:59:36.526887 2231 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:36.521421 2281 raft_consensus.cc:2747] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250831 07:59:36.533560 32462 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38366 (request call id 502) took 2512 ms. Trace:
I20250831 07:59:36.506412 2287 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.329s
I20250831 07:59:36.528685 2270 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:36.524289 2208 raft_consensus.cc:2747] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:36.498203 2166 consensus_queue.cc:1046] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.046s
I20250831 07:59:36.501349 2205 consensus_queue.cc:1046] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20250831 07:59:36.546825 2264 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.591810 2264 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:36.600040 2256 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:36.572331 2298 log.cc:927] Time spent T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
I20250831 07:59:36.573659 2310 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:36.581161 31801 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:36.607806 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:36.616499 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:36.617115 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:36.516526 2262 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:36.640488 31773 leader_election.cc:336] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.693s (SENT)
I20250831 07:59:36.640993 31773 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:36.606504 2205 consensus_queue.cc:1230] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer 7512ba2ed3a94982a22a75f51a647e65 log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250831 07:59:36.642572 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:36.644007 31729 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:36.651676 31877 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
W20250831 07:59:36.637982 2293 log.cc:927] Time spent T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.356s user 0.000s sys 0.001s
W20250831 07:59:36.364508 1962 consensus_queue.cc:1162] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8247a38b550649aa9cd6e7668dcaec10" responder_term: 6 status { last_received { term: 3 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: [5.4-5.4]" } } last_received_current_leader { term: 0 index: 0 } }
I20250831 07:59:36.582945 2270 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:36.653213 2249 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:36.767175 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:36.767804 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:36.769398 31877 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:36.790627 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:36.637166 2310 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:36.746404 2269 consensus_meta.cc:287] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.540s user 0.009s sys 0.002s
I20250831 07:59:36.689894 2302 raft_consensus.cc:2747] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:36.738381 2280 raft_consensus.cc:2747] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:36.790016 2258 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:36.787308 2284 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:36.813373 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1821) took 1118 ms. Trace:
W20250831 07:59:36.800441 2201 consensus_meta.cc:287] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.723s user 0.011s sys 0.000s
I20250831 07:59:36.743963 2277 raft_consensus.cc:2747] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:36.815258 31883 rpcz_store.cc:276] 0831 07:59:35.695326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:35.695538 (+ 212us) service_pool.cc:225] Handling call
0831 07:59:35.696020 (+ 482us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:35.715440 (+ 19420us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:35.715487 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:35.716131 (+ 644us) log.cc:844] Serialized 26 byte log entry
0831 07:59:36.073239 (+357108us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:36.073278 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:36.073342 (+ 64us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:36.813106 (+739764us) raft_consensus.cc:1683] finished
0831 07:59:36.813135 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:36.813298 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12660,"raft.queue_time_us":36441,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":6017,"thread_start_us":356623,"threads_started":1,"wal-append.queue_time_us":464786}
I20250831 07:59:36.821314 31883 raft_consensus.cc:1238] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:36.831100 31883 raft_consensus.cc:1238] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250831 07:59:36.809741 2125 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:36.858590 31730 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.007s sys 0.022s
W20250831 07:59:36.876773 31730 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.683s user 0.007s sys 0.025s
I20250831 07:59:36.877561 31801 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:36.737138 2249 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:36.832473 1740 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:36.809495 2269 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.833808 2157 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:36.843957 2166 consensus_queue.cc:1230] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer 7512ba2ed3a94982a22a75f51a647e65 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250831 07:59:36.917541 2270 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:36.570451 32462 rpcz_store.cc:276] 0831 07:59:34.021039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:34.138168 (+117129us) service_pool.cc:225] Handling call
0831 07:59:34.156267 (+ 18099us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:34.359401 (+203134us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:34.359434 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:34.442701 (+ 83267us) log.cc:844] Serialized 26 byte log entry
0831 07:59:34.564881 (+122180us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:34.569571 (+ 4690us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:35.555358 (+985787us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b540074e354. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:35.555398 (+ 40us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:36.348963 (+793565us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:36.520583 (+171620us) raft_consensus.cc:1683] finished
0831 07:59:36.522429 (+ 1846us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:36.533347 (+ 10918us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25768,"raft.queue_time_us":27714,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":2225,"spinlock_wait_cycles":374856832,"thread_start_us":62066,"threads_started":1,"wal-append.queue_time_us":169736}
I20250831 07:59:36.918751 2146 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:36.919203 2104 consensus_meta.cc:287] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.757s user 0.009s sys 0.004s
W20250831 07:59:36.921833 2099 consensus_meta.cc:287] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.946s user 0.000s sys 0.010s
I20250831 07:59:36.922259 2099 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:36.922966 2204 raft_consensus.cc:3053] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:36.923398 31802 raft_consensus.cc:1238] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: []
W20250831 07:59:36.856256 2250 consensus_meta.cc:287] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.943s user 0.000s sys 0.012s
W20250831 07:59:36.856642 2308 log.cc:927] Time spent T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
W20250831 07:59:36.857635 2299 log.cc:927] Time spent T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.287s user 0.000s sys 0.001s
I20250831 07:59:36.838418 2264 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:36.826421 2284 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:36.901831 2262 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:36.923507 2204 raft_consensus.cc:738] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
W20250831 07:59:36.952955 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:36.959582 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:36.961225 32434 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:36.928838 2225 consensus_queue.cc:1046] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
W20250831 07:59:36.968919 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:36.971678 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 329435us
I20250831 07:59:36.979157 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:36.988132 31731 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250831 07:59:36.916671 1740 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.945367 2264 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:36.999372 31731 raft_consensus.cc:1238] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
W20250831 07:59:37.006295 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:36.936033 2250 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:37.082705 31726 raft_consensus.cc:1238] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:36.989651 2262 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.082770 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:36.989883 2220 raft_consensus.cc:3053] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 9 LEADER]: Stepping down as leader of term 9
W20250831 07:59:36.998940 2306 log.cc:927] Time spent T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.206s user 0.000s sys 0.006s
I20250831 07:59:36.997203 2166 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:37.012639 2310 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.081993 2319 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:37.156481 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:37.083285 2259 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.083089 2244 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:37.236044 2244 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.238319 2244 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.088668 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:37.196606 31806 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8. Ops: [7.6-7.6]
W20250831 07:59:37.226253 2209 log.cc:927] Time spent T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 2.063s user 0.000s sys 0.005s
W20250831 07:59:37.244609 2321 log.cc:927] Time spent T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.246s user 0.000s sys 0.001s
I20250831 07:59:36.994695 2204 consensus_queue.cc:260] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.226569 2330 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:37.227041 2289 log.cc:927] Time spent T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20250831 07:59:37.232167 2259 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:37.165364 2308 log.cc:927] Time spent T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20250831 07:59:37.153084 2220 raft_consensus.cc:738] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:37.261605 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1919) took 1064 ms. Trace:
I20250831 07:59:37.277494 31729 rpcz_store.cc:276] 0831 07:59:36.196879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.196993 (+ 114us) service_pool.cc:225] Handling call
0831 07:59:37.261338 (+1064345us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b5400311d54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:37.261498 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":764743552}
W20250831 07:59:37.262746 31882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1427) took 1476 ms (client timeout 1693 ms). Trace:
W20250831 07:59:37.278559 31882 rpcz_store.cc:269] 0831 07:59:35.786277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:35.786411 (+ 134us) service_pool.cc:225] Handling call
0831 07:59:37.262480 (+1476069us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b5400299fd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:37.262644 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":898787072}
I20250831 07:59:37.264690 1947 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:37.269373 2231 consensus_meta.cc:287] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.694s user 0.008s sys 0.002s
I20250831 07:59:37.155907 2270 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:37.281148 2342 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.281548 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:37.281746 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 292) took 1101 ms. Trace:
I20250831 07:59:37.282295 2220 consensus_queue.cc:260] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.157859 2262 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.283303 2208 consensus_queue.cc:1046] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 07:59:37.197753 2099 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:37.211206 2249 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:37.291850 31882 raft_consensus.cc:1238] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
W20250831 07:59:37.276355 2251 consensus_meta.cc:287] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 1.142s user 0.009s sys 0.003s
I20250831 07:59:37.326288 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1446) took 1142 ms. Trace:
I20250831 07:59:37.306639 2270 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.326633 31882 rpcz_store.cc:276] 0831 07:59:36.183435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:37.288995 (+1105560us) service_pool.cc:225] Handling call
0831 07:59:37.289307 (+ 312us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:37.326010 (+ 36703us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:37.326236 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:37.276428 2269 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.307914 31728 rpcz_store.cc:276] 0831 07:59:36.180420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.180952 (+ 532us) service_pool.cc:225] Handling call
0831 07:59:36.181238 (+ 286us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:36.198563 (+ 17325us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:36.198596 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:36.199029 (+ 433us) log.cc:844] Serialized 26 byte log entry
0831 07:59:36.496186 (+297157us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:36.496231 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:36.496318 (+ 87us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:37.281524 (+785206us) raft_consensus.cc:1683] finished
0831 07:59:37.281547 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:37.281690 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57772,"raft.queue_time_us":6974,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":8783,"thread_start_us":198236,"threads_started":1,"wal-append.queue_time_us":432322}
I20250831 07:59:37.312261 2342 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.276808 2330 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:37.321787 2336 log.cc:927] Time spent T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.069s user 0.000s sys 0.001s
I20250831 07:59:37.331480 1740 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.330291 2284 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:37.323540 2137 consensus_meta.cc:287] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 1.154s user 0.013s sys 0.000s
I20250831 07:59:37.328811 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 291) took 1162 ms. Trace:
I20250831 07:59:37.324541 2249 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:37.344960 2249 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:37.345825 2236 consensus_meta.cc:287] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.849s user 0.004s sys 0.006s
I20250831 07:59:37.307314 2157 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.347783 2270 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.348122 2157 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.349886 2330 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:37.350476 2157 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.351524 2250 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:37.343446 2284 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:37.352159 2284 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.335723 2251 raft_consensus.cc:2747] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:37.353376 2157 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:37.354035 2157 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:37.353495 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:37.355623 2307 log.cc:927] Time spent T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.773s user 0.006s sys 0.004s
I20250831 07:59:37.347452 2249 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:37.357201 2137 raft_consensus.cc:2747] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:37.356272 2157 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:37.358129 2284 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:37.353466 2259 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:37.358862 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:37.359519 31802 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:37.342314 31730 rpcz_store.cc:276] 0831 07:59:36.180099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.180359 (+ 260us) service_pool.cc:225] Handling call
0831 07:59:36.180613 (+ 254us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:36.180795 (+ 182us) raft_consensus.cc:1532] Early marking committed up to index 6
0831 07:59:36.879964 (+699169us) log.cc:844] Serialized 14 byte log entry
0831 07:59:37.328020 (+448056us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:37.328092 (+ 72us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:37.328163 (+ 71us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:37.328253 (+ 90us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:37.328405 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40854,"prepare.queue_time_us":698233,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":45104,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":6323,"thread_start_us":1090754,"threads_started":2}
I20250831 07:59:37.363245 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1721) took 1066 ms. Trace:
I20250831 07:59:37.363718 31804 rpcz_store.cc:276] 0831 07:59:36.296985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.297369 (+ 384us) service_pool.cc:225] Handling call
0831 07:59:36.297754 (+ 385us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:36.298156 (+ 402us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:36.298197 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:36.317604 (+ 19407us) log.cc:844] Serialized 26 byte log entry
0831 07:59:36.459349 (+141745us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:36.685270 (+225921us) log.cc:844] Serialized 14 byte log entry
0831 07:59:36.685550 (+ 280us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:36.685717 (+ 167us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:37.354965 (+669248us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:37.362924 (+ 7959us) raft_consensus.cc:1683] finished
0831 07:59:37.363032 (+ 108us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:37.363185 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":473774,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":202707,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":18557,"spinlock_wait_cycles":43481856,"thread_start_us":308903,"threads_started":2,"wal-append.queue_time_us":186268}
I20250831 07:59:37.365805 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:37.380342 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1867) took 1564 ms. Trace:
I20250831 07:59:37.293195 2231 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
W20250831 07:59:37.388836 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 420007us
I20250831 07:59:37.390309 2104 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:37.393779 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:37.253552 2319 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.410449 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:37.380870 31805 rpcz_store.cc:276] 0831 07:59:35.815756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:35.815999 (+ 243us) service_pool.cc:225] Handling call
0831 07:59:35.816440 (+ 441us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:35.836490 (+ 20050us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 07:59:35.837339 (+ 849us) log.cc:844] Serialized 14 byte log entry
0831 07:59:36.136103 (+298764us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b18002604a0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b792 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:59:37.370893 (+1234790us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b18002604a0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce7c4955c 00007fdce507e77b 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce5098805 00007fdce7ff5020 00007fdce80076c6 00007fdce8018237
0831 07:59:37.371100 (+ 207us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:37.371149 (+ 49us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:37.371194 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:37.371813 (+ 619us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b54000857d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:37.380114 (+ 8301us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:37.380271 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23440,"prepare.queue_time_us":26448,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":6080,"raft.queue_time_us":20047,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"spinlock_wait_cycles":77584000,"thread_start_us":75490,"threads_started":1,"wal-append.queue_time_us":6501,"wal-append.run_cpu_time_us":1089,"wal-append.run_wall_time_us":1094744}
I20250831 07:59:37.354185 2264 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.419229 31725 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20250831 07:59:37.394711 2104 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.438144 2104 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.438705 2319 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:37.439265 2325 log.cc:927] Time spent T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.186s user 0.000s sys 0.000s
I20250831 07:59:37.441679 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1720) took 1145 ms. Trace:
I20250831 07:59:37.431267 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:37.391047 2278 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:37.442883 2342 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.617339 2244 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:37.445168 31802 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:37.501504 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:37.478662 31698 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:37.468874 31804 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:37.412392 2223 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.624044 31805 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8.
I20250831 07:59:37.631101 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:37.630491 31775 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.034s sys 0.035s
W20250831 07:59:37.632545 31775 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.633s user 0.035s sys 0.035s
W20250831 07:59:37.624584 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 146056us
I20250831 07:59:37.620304 31728 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:37.442162 31809 rpcz_store.cc:276] 0831 07:59:36.296550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.296697 (+ 147us) service_pool.cc:225] Handling call
0831 07:59:36.297032 (+ 335us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 07:59:36.490044 (+193012us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:36.490106 (+ 62us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 07:59:36.490887 (+ 781us) log.cc:844] Serialized 52 byte log entry
0831 07:59:36.733818 (+242931us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:36.959190 (+225372us) log.cc:844] Serialized 14 byte log entry
0831 07:59:36.976867 (+ 17677us) log.cc:844] Serialized 14 byte log entry
0831 07:59:36.993538 (+ 16671us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:36.993680 (+ 142us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:37.372561 (+378881us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b18003b13a0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b8bf 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:59:37.441382 (+ 68821us) raft_consensus.cc:1683] finished
0831 07:59:37.441423 (+ 41us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:37.441607 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":142444,"prepare.queue_time_us":373225,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":4560,"raft.queue_time_us":81427,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":74050,"spinlock_wait_cycles":39739264,"thread_start_us":563294,"threads_started":3,"wal-append.queue_time_us":501215}
W20250831 07:59:37.632982 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 633268us
I20250831 07:59:37.417763 2259 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.477640 1740 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:37.593358 2264 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.681478 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 407) took 1212 ms. Trace:
I20250831 07:59:37.690380 31802 rpcz_store.cc:276] 0831 07:59:36.468717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:37.680553 (+1211836us) service_pool.cc:225] Handling call
0831 07:59:37.680918 (+ 365us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:37.681114 (+ 196us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:37.681142 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:37.681191 (+ 49us) raft_consensus.cc:1643] Marking committed up to 9
0831 07:59:37.681227 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:37.681296 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:37.681419 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:37.691860 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:37.683247 2343 log.cc:927] Time spent T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250831 07:59:37.506065 2104 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.621028 2247 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:37.688612 32435 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:37.545930 1804 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:37.691905 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:37.705194 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:37.705876 31804 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:37.617478 2231 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:37.494346 2269 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:37.540894 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:37.697247 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 408) took 1227 ms. Trace:
I20250831 07:59:37.741348 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:37.745570 31809 rpcz_store.cc:276] 0831 07:59:36.470222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:37.684758 (+1214536us) service_pool.cc:225] Handling call
0831 07:59:37.696591 (+ 11833us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:37.696813 (+ 222us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:37.696849 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:37.696926 (+ 77us) raft_consensus.cc:1643] Marking committed up to 8
0831 07:59:37.696972 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:37.697043 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:37.697185 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:37.726687 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 300) took 1517 ms. Trace:
I20250831 07:59:37.692834 31802 raft_consensus.cc:2391] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 4: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:37.629784 2270 raft_consensus.cc:2747] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:37.754964 2270 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:37.755501 2270 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:37.756292 2271 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:37.756948 2271 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.757495 2270 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:37.759398 2271 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:37.709565 31877 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.005s sys 0.018s
I20250831 07:59:37.716579 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:37.631071 2262 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:37.501194 2127 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:37.767021 31730 raft_consensus.cc:2391] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:37.696291 2340 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:37.775696 31874 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.024s sys 0.072s
W20250831 07:59:37.777072 31874 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s user 0.025s sys 0.072s
I20250831 07:59:37.752013 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:37.748008 31882 rpcz_store.cc:276] 0831 07:59:36.209514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:37.327487 (+1117973us) service_pool.cc:225] Handling call
0831 07:59:37.327848 (+ 361us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:37.328088 (+ 240us) raft_consensus.cc:1532] Early marking committed up to index 6
0831 07:59:37.333200 (+ 5112us) log.cc:844] Serialized 14 byte log entry
0831 07:59:37.726249 (+393049us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:37.726321 (+ 72us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:37.726381 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:37.726472 (+ 91us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:37.726619 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":5406,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":38,"thread_start_us":392631,"threads_started":1}
I20250831 07:59:37.741564 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:37.780072 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:37.780354 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:37.709254 2104 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:37.707165 31849 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.019s sys 0.026s
I20250831 07:59:37.762013 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:37.501633 2251 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.540326 2284 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.698909 2342 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:37.695940 2259 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.700410 2264 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:37.767755 31877 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.646s user 0.006s sys 0.018s
I20250831 07:59:37.723984 2250 consensus_queue.cc:1046] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.061s
I20250831 07:59:37.774714 31804 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:37.725342 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1441) took 1593 ms. Trace:
I20250831 07:59:37.806234 31878 rpcz_store.cc:276] 0831 07:59:36.132125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.166191 (+ 34066us) service_pool.cc:225] Handling call
0831 07:59:37.725123 (+1558932us) spinlock_profiling.cc:243] Waited 96.6 ms on lock 0x7b5400247054. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:37.725269 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":222148480}
I20250831 07:59:37.777376 2127 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.804767 2351 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:37.807067 2341 consensus_queue.cc:1046] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.112s
I20250831 07:59:37.807294 2351 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.808022 2341 raft_consensus.cc:3053] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Stepping down as leader of term 7
I20250831 07:59:37.808388 2341 raft_consensus.cc:738] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:37.808526 2127 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.809159 2341 consensus_queue.cc:260] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.709424 2146 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.810380 2104 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.810863 2146 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.811317 2349 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.811842 2349 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.812938 2146 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.685333 2244 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.813818 2349 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.760185 2326 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:37.781433 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:37.817238 31730 raft_consensus.cc:2391] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 10: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
W20250831 07:59:37.784723 31849 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.677s user 0.019s sys 0.026s
W20250831 07:59:37.818051 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 710366us
W20250831 07:59:37.818151 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 198387us
I20250831 07:59:37.820163 32436 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:37.793108 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 301) took 1583 ms. Trace:
I20250831 07:59:37.675949 2223 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:37.767717 31848 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.018s sys 0.037s
W20250831 07:59:37.822628 31848 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.681s user 0.018s sys 0.037s
I20250831 07:59:37.762887 2269 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.696090 1740 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.723456 2157 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.068s
I20250831 07:59:37.810317 2341 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:37.820851 31882 rpcz_store.cc:276] 0831 07:59:36.209715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:37.792289 (+1582574us) service_pool.cc:225] Handling call
0831 07:59:37.792622 (+ 333us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:37.792803 (+ 181us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:37.792825 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:37.792866 (+ 41us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:37.792904 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:37.792955 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:37.793056 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:37.726740 1804 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.633342 2278 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:37.793787 2255 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:37.831745 31729 raft_consensus.cc:1273] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:37.833830 31848 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:37.837210 31847 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.012s sys 0.025s
W20250831 07:59:37.843263 31847 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.573s user 0.013s sys 0.026s
W20250831 07:59:37.826522 2226 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.005s sys 0.030s
I20250831 07:59:37.816552 2244 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:37.837733 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 152157us
I20250831 07:59:37.805625 2342 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:37.888062 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 54310.6us
I20250831 07:59:37.888922 31848 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:37.709478 2247 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.803853 2284 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.901891 2342 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.903810 1804 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.904644 2355 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:37.905161 2355 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:37.907110 2355 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.803421 2251 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:37.913395 2251 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:37.784730 2340 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.915122 31726 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:37.915988 2340 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.918329 2247 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:37.918453 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:37.889360 2262 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.919203 2262 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.903553 2264 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.921109 2262 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:37.922385 2262 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:37.922971 2262 raft_consensus.cc:2747] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 07:59:37.925155 1740 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.926677 2278 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:37.829295 2326 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:37.928793 2326 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:37.881886 2226 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s user 0.005s sys 0.031s
I20250831 07:59:37.933099 2259 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:37.933645 2259 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.934087 2259 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:37.940107 2344 log.cc:927] Time spent T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20250831 07:59:37.945116 2254 log.cc:927] Time spent T a187e6c5ed8f4214b4486aab25955909 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 2.197s user 0.003s sys 0.000s
I20250831 07:59:37.948729 2096 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 07:59:37.949306 2096 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:37.949723 2096 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:37.951923 2186 raft_consensus.cc:3053] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:37.952301 2186 raft_consensus.cc:738] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:37.956048 1966 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:37.956719 1966 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:37.958879 1966 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:37.961393 31801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1727) took 1568 ms (client timeout 1691 ms). Trace:
I20250831 07:59:37.961628 31804 raft_consensus.cc:1238] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
W20250831 07:59:37.961805 31801 rpcz_store.cc:269] 0831 07:59:36.393085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.393339 (+ 254us) service_pool.cc:225] Handling call
0831 07:59:37.961185 (+1567846us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54000cf3d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:37.961335 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":363074816}
I20250831 07:59:37.963310 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:37.982756 31801 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:37.963902 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:37.985234 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1719) took 1692 ms. Trace:
W20250831 07:59:37.965962 2312 log.cc:927] Time spent T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 1.381s user 0.002s sys 0.000s
I20250831 07:59:37.966725 2269 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:37.969676 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:37.971117 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 295) took 2135 ms. Trace:
I20250831 07:59:37.994385 31874 rpcz_store.cc:276] 0831 07:59:35.835299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:35.838251 (+ 2952us) service_pool.cc:225] Handling call
0831 07:59:35.838575 (+ 324us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 07:59:35.838910 (+ 335us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:35.838940 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 07:59:35.839482 (+ 542us) log.cc:844] Serialized 52 byte log entry
0831 07:59:37.245225 (+1405743us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b18003aa5c0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce507e0f7 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdce7c3b2bb 00007fdce7c40dc3 00007fdce7c41171 00007fdce7fd009f
0831 07:59:37.782108 (+536883us) raft_consensus.cc:1643] Marking committed up to 8
0831 07:59:37.805112 (+ 23004us) log.cc:844] Serialized 14 byte log entry
0831 07:59:37.815105 (+ 9993us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:37.815204 (+ 99us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:37.970748 (+155544us) raft_consensus.cc:1683] finished
0831 07:59:37.970850 (+ 102us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:37.971024 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9616,"prepare.queue_time_us":22771,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":9810,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":57478,"spinlock_wait_cycles":5379840,"thread_start_us":536636,"threads_started":1,"wal-append.queue_time_us":2107456}
W20250831 07:59:37.972503 2347 log.cc:927] Time spent T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250831 07:59:37.977921 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1912) took 1783 ms. Trace:
I20250831 07:59:37.887377 2255 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:38.011178 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:38.012193 2255 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:37.977936 2223 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:37.985767 31806 rpcz_store.cc:276] 0831 07:59:36.292623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.292813 (+ 190us) service_pool.cc:225] Handling call
0831 07:59:36.293240 (+ 427us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:37.241452 (+948212us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:37.984927 (+743475us) spinlock_profiling.cc:243] Waited 871 ms on lock 0x7b54002f0654. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:37.985164 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2003059968}
W20250831 07:59:38.016973 2309 log.cc:927] Time spent T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 1.444s user 0.003s sys 0.001s
I20250831 07:59:38.021428 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1914) took 1826 ms. Trace:
I20250831 07:59:38.018976 2241 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:38.024248 2241 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:38.026844 2241 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:37.840668 2352 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:37.991299 31875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1829) took 1833 ms (client timeout 1555 ms). Trace:
I20250831 07:59:37.907599 2355 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:37.923442 2262 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:37.998703 31804 raft_consensus.cc:1273] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 07:59:38.001524 31732 rpcz_store.cc:276] 0831 07:59:36.194145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.194296 (+ 151us) service_pool.cc:225] Handling call
0831 07:59:36.216463 (+ 22167us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:36.516495 (+300032us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:37.977629 (+1461134us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b54001aac54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:37.977846 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":687907072}
I20250831 07:59:38.003892 2092 consensus_queue.cc:260] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:38.005725 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 405) took 1566 ms. Trace:
I20250831 07:59:38.052255 31805 rpcz_store.cc:276] 0831 07:59:36.439638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:37.632941 (+1193303us) service_pool.cc:225] Handling call
0831 07:59:37.684373 (+ 51432us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:37.895933 (+211560us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:37.895964 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:37.896009 (+ 45us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:37.896049 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:38.005499 (+109450us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b540009f454. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:38.005543 (+ 44us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:38.005662 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":486024064}
W20250831 07:59:38.049698 31875 rpcz_store.cc:269] 0831 07:59:36.157910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:37.990801 (+1832891us) service_pool.cc:212] Skipping call since client already timed out
0831 07:59:37.991213 (+ 412us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 07:59:38.009739 2284 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:37.992746 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.031126 31880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 296) took 2192 ms. Trace:
I20250831 07:59:38.091053 31801 raft_consensus.cc:2408] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 07:59:38.034119 31806 raft_consensus.cc:1238] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:38.029748 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:38.021924 31733 rpcz_store.cc:276] 0831 07:59:36.194864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.194990 (+ 126us) service_pool.cc:225] Handling call
0831 07:59:36.195314 (+ 324us) raft_consensus.cc:1469] Updating replica for 3 ops
0831 07:59:36.205095 (+ 9781us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:36.205128 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 3 ops
0831 07:59:36.205798 (+ 670us) log.cc:844] Serialized 78 byte log entry
0831 07:59:36.435938 (+230140us) raft_consensus.cc:1643] Marking committed up to 8
0831 07:59:36.733187 (+297249us) log.cc:844] Serialized 14 byte log entry
0831 07:59:36.739962 (+ 6775us) log.cc:844] Serialized 14 byte log entry
0831 07:59:36.740291 (+ 329us) log.cc:844] Serialized 14 byte log entry
0831 07:59:36.798366 (+ 58075us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:36.798463 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:37.243203 (+444740us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b180048b320. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b8bf 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:59:38.016223 (+773020us) spinlock_profiling.cc:243] Waited 28.8 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:38.021124 (+ 4901us) raft_consensus.cc:1683] finished
0831 07:59:38.021155 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:38.021340 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":148008,"prepare.queue_time_us":1085643,"prepare.run_cpu_time_us":178,"prepare.run_wall_time_us":20718,"raft.queue_time_us":98274,"raft.run_cpu_time_us":600,"raft.run_wall_time_us":106328,"spinlock_wait_cycles":71538176,"thread_start_us":489476,"threads_started":2,"wal-append.queue_time_us":366929}
W20250831 07:59:38.054711 31698 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.555s (SENT) [suppressed 48 similar messages]
I20250831 07:59:38.095940 32470 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 8: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
W20250831 07:59:38.058462 31727 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.742s user 0.010s sys 0.026s
I20250831 07:59:38.090520 31880 rpcz_store.cc:276] 0831 07:59:35.838447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:35.838570 (+ 123us) service_pool.cc:225] Handling call
0831 07:59:35.861996 (+ 23426us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:36.068381 (+206385us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b1800266aa0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce7c4955c 00007fdce507e77b 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce5098805 00007fdce7ff5020 00007fdce7ff5bbb 00007fdce80005fb
0831 07:59:36.068518 (+ 137us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:36.068543 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:36.069011 (+ 468us) log.cc:844] Serialized 26 byte log entry
0831 07:59:36.446745 (+377734us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:36.668432 (+221687us) log.cc:844] Serialized 14 byte log entry
0831 07:59:36.668846 (+ 414us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:36.668965 (+ 119us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:37.965101 (+1296136us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:38.006322 (+ 41221us) raft_consensus.cc:1683] finished
0831 07:59:38.006344 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:38.031065 (+ 24721us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12964,"prepare.queue_time_us":163326,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":12010,"raft.queue_time_us":35557,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":30283,"spinlock_wait_cycles":41661056,"thread_start_us":597064,"threads_started":3,"wal-append.queue_time_us":515681}
W20250831 07:59:38.098438 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:38.098536 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:38.106616 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 418070us
W20250831 07:59:38.096941 31727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.780s user 0.010s sys 0.026s
I20250831 07:59:38.108577 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:38.035208 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:38.117486 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:37.999157 2326 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:38.122202 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.123347 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:38.123420 31801 raft_consensus.cc:1215] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Deduplicated request from leader. Original: 4.6->[6.7-6.7] Dedup: 6.7->[]
I20250831 07:59:38.124260 31773 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:38.124907 31801 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250831 07:59:38.125833 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.126761 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:38.027721 2351 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:38.135221 31732 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250831 07:59:38.061420 2259 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:38.135885 2146 raft_consensus.cc:2747] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 07:59:38.127211 2258 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:38.153734 2258 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:38.061209 2352 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:38.127911 2319 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:38.159619 2319 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:38.127537 2284 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:38.164330 2284 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:38.166711 2284 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:38.117273 2242 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:38.168557 2242 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:38.170195 2315 log.cc:927] Time spent T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 1.425s user 0.000s sys 0.003s
I20250831 07:59:38.171000 2242 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:38.136596 2326 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:38.173532 2291 log.cc:927] Time spent T f611ea44f17c403e818574844e13b946 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 1.720s user 0.001s sys 0.001s
I20250831 07:59:38.127635 2342 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:38.174998 2326 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:38.175729 2342 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:38.176831 2352 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:38.177820 2342 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:38.178428 31803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39068 (request call id 712) took 2024 ms. Trace:
I20250831 07:59:38.178807 31803 rpcz_store.cc:276] 0831 07:59:36.153893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.154016 (+ 123us) service_pool.cc:225] Handling call
0831 07:59:38.177784 (+2023768us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54002f0654. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f81cd0 00007fdce8005eea 00007fdcecf5e60e 00007fdce65fbcc7 00007fdce65fbc52 00007fdce65fbb81 00007fdce65fbafc 00007fdce65fadb2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:38.178337 (+ 553us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2321105792}
I20250831 07:59:38.117766 2241 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:38.117053 2124 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:38.141731 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.138702 31773 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:38.060424 2262 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:38.162869 2258 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:38.196692 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1440) took 2065 ms. Trace:
I20250831 07:59:38.202769 31881 rpcz_store.cc:276] 0831 07:59:36.131789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.178788 (+ 46999us) service_pool.cc:225] Handling call
0831 07:59:36.179136 (+ 348us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:36.180181 (+ 1045us) raft_consensus.cc:1532] Early marking committed up to index 6
0831 07:59:36.366588 (+186407us) log.cc:844] Serialized 14 byte log entry
0831 07:59:36.556708 (+190120us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:36.613607 (+ 56899us) log.cc:844] Serialized 26 byte log entry
0831 07:59:36.614269 (+ 662us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:36.615090 (+ 821us) log.cc:844] Serialized 14 byte log entry
0831 07:59:36.615399 (+ 309us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:36.615470 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:38.169548 (+1554078us) spinlock_profiling.cc:243] Waited 43.6 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:38.196428 (+ 26880us) raft_consensus.cc:1683] finished
0831 07:59:38.196459 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:38.196622 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34574,"prepare.queue_time_us":379457,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":2920,"raft.queue_time_us":11432,"raft.run_cpu_time_us":393,"raft.run_wall_time_us":43338,"spinlock_wait_cycles":100870912,"thread_start_us":280820,"threads_started":2,"wal-append.queue_time_us":258769}
I20250831 07:59:38.204836 2262 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:38.162058 2319 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:38.135526 32435 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:38.235363 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 96744.3us
I20250831 07:59:38.253042 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:38.194438 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:38.284787 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.283250 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.252761 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.288410 31801 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:38.289026 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.289700 31803 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:38.289700 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.293660 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.294354 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.296769 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.297576 31806 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:38.298398 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.296614 31728 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:38.292265 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.302759 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.303586 31730 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
W20250831 07:59:38.216301 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 80857.3us
I20250831 07:59:38.307186 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.310433 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.311233 31728 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:38.318852 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.307869 31730 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:38.327382 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.328188 31806 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:38.283036 2319 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:38.285624 2337 raft_consensus.cc:2802] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 07:59:38.332377 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 443542us
I20250831 07:59:38.332455 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.340826 31801 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:38.335209 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.335724 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.342229 31730 raft_consensus.cc:2391] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:38.336786 31849 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:38.337412 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.284369 2362 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20250831 07:59:38.329401 2104 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.007s sys 0.028s
I20250831 07:59:38.214804 2124 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:38.353139 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.352600 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.212121 2241 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:38.329638 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:38.473902 31847 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.003s sys 0.038s
I20250831 07:59:38.713598 31731 raft_consensus.cc:1273] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:38.441090 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.715441 31730 raft_consensus.cc:2466] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
W20250831 07:59:38.455591 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 148055us
W20250831 07:59:38.716006 31882 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.016s sys 0.004s
W20250831 07:59:38.716440 31882 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s user 0.016s sys 0.004s
I20250831 07:59:38.357023 2259 raft_consensus.cc:2802] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election won for term 6
W20250831 07:59:38.356122 2104 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s user 0.007s sys 0.028s
W20250831 07:59:38.603799 31698 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.008s sys 0.019s
W20250831 07:59:38.748276 31698 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.653s user 0.013s sys 0.020s
W20250831 07:59:38.748726 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 694018us
W20250831 07:59:38.474459 31878 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.019s sys 0.022s
W20250831 07:59:38.752909 31878 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.867s user 0.019s sys 0.036s
W20250831 07:59:38.713739 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 364451us
I20250831 07:59:38.754232 31848 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:38.456553 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.720607 2346 consensus_queue.cc:1046] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.7, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250831 07:59:38.340559 2277 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:59:38.402763 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:38.523591 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:38.755445 31734 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 8.
W20250831 07:59:38.426893 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:38.763556 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:38.763672 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.713382 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:38.370913 31877 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.018s sys 0.033s
I20250831 07:59:38.764605 31801 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Advancing to term 9
W20250831 07:59:38.744441 31804 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.012s sys 0.028s
W20250831 07:59:38.371891 31809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.024s sys 0.042s
I20250831 07:59:38.746412 31730 raft_consensus.cc:1238] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
W20250831 07:59:38.330478 2272 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.759s user 0.022s sys 0.026s
I20250831 07:59:38.747303 31726 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:38.341651 2146 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:38.756352 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:38.757994 31699 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:38.346417 2319 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:38.714402 31847 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.803s user 0.014s sys 0.041s
I20250831 07:59:38.368669 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:38.769711 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:38.621145 2135 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.009s sys 0.031s
I20250831 07:59:38.769941 31728 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
W20250831 07:59:38.764660 31877 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.959s user 0.026s sys 0.043s
I20250831 07:59:38.347342 2337 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:38.765725 31804 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s user 0.012s sys 0.028s
W20250831 07:59:38.765834 31809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.967s user 0.040s sys 0.044s
I20250831 07:59:38.771533 31877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1442) took 2627 ms. Trace:
W20250831 07:59:38.588344 2182 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.017s sys 0.023s
I20250831 07:59:38.773062 31805 raft_consensus.cc:1215] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 LEADER]: Deduplicated request from leader. Original: 6.7->[8.8-8.8] Dedup: 8.8->[]
I20250831 07:59:38.782581 31805 raft_consensus.cc:1238] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 LEADER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20250831 07:59:38.775203 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:38.783620 2346 consensus_queue.cc:1230] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer 7512ba2ed3a94982a22a75f51a647e65 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250831 07:59:38.779657 2146 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:38.786113 2146 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:38.787647 2376 raft_consensus.cc:2802] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:38.707019 2348 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:38.802592 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 409) took 1089 ms. Trace:
I20250831 07:59:38.803361 31804 rpcz_store.cc:276] 0831 07:59:37.713281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:38.050882 (+337601us) service_pool.cc:225] Handling call
0831 07:59:38.051216 (+ 334us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:38.051418 (+ 202us) raft_consensus.cc:1532] Early marking committed up to index 9
0831 07:59:38.099544 (+ 48126us) log.cc:844] Serialized 14 byte log entry
0831 07:59:38.802091 (+702547us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:38.802174 (+ 83us) raft_consensus.cc:1643] Marking committed up to 9
0831 07:59:38.802257 (+ 83us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:38.802354 (+ 97us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:38.802516 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22435,"prepare.queue_time_us":47958,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":9284,"thread_start_us":679673,"threads_started":1}
W20250831 07:59:38.774629 2272 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.203s user 0.026s sys 0.026s
W20250831 07:59:38.775849 2135 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.803s user 0.011s sys 0.038s
I20250831 07:59:38.788136 2376 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:38.779983 31877 rpcz_store.cc:276] 0831 07:59:36.144140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.808905 (+664765us) service_pool.cc:225] Handling call
0831 07:59:36.809243 (+ 338us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:36.811195 (+ 1952us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 07:59:36.828054 (+ 16859us) log.cc:844] Serialized 14 byte log entry
0831 07:59:37.804416 (+976362us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:37.804949 (+ 533us) log.cc:844] Serialized 26 byte log entry
0831 07:59:37.805137 (+ 188us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:38.770900 (+965763us) log.cc:844] Serialized 14 byte log entry
0831 07:59:38.771128 (+ 228us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:38.771228 (+ 100us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:38.771298 (+ 70us) raft_consensus.cc:1683] finished
0831 07:59:38.771317 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:38.771467 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34677,"prepare.queue_time_us":23946,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":76,"raft.queue_time_us":213053,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":5753,"thread_start_us":1879562,"threads_started":3,"wal-append.queue_time_us":563099}
W20250831 07:59:38.789239 2182 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.820s user 0.020s sys 0.027s
I20250831 07:59:38.812813 32464 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8. Ops: []
I20250831 07:59:38.814873 2124 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:38.816273 2341 raft_consensus.cc:2762] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20250831 07:59:38.816311 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:38.358382 2332 consensus_queue.cc:1035] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.434s
I20250831 07:59:38.781198 2366 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:38.777487 2262 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:38.820881 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:38.573916 2352 consensus_queue.cc:1046] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.229s
I20250831 07:59:38.834780 31801 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 9.
W20250831 07:59:38.836537 31883 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.018s sys 0.023s
W20250831 07:59:38.856765 31883 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.018s sys 0.023s
I20250831 07:59:38.857754 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1445) took 2674 ms. Trace:
I20250831 07:59:38.858263 31883 rpcz_store.cc:276] 0831 07:59:36.183139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.837389 (+654250us) service_pool.cc:225] Handling call
0831 07:59:36.840009 (+ 2620us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:36.862184 (+ 22175us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:36.862252 (+ 68us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:36.892172 (+ 29920us) log.cc:844] Serialized 26 byte log entry
0831 07:59:37.150665 (+258493us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:37.159391 (+ 8726us) log.cc:844] Serialized 14 byte log entry
0831 07:59:38.203276 (+1043885us) spinlock_profiling.cc:243] Waited 34.9 ms on lock 0x7b18003200e0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce507e0f7 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce5098805 00007fdce7ff5020 00007fdce80076c6 00007fdce8018237 00007fdce80181b8
0831 07:59:38.857264 (+653988us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:38.857374 (+ 110us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:38.857437 (+ 63us) raft_consensus.cc:1683] finished
0831 07:59:38.857532 (+ 95us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:38.857686 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4139,"prepare.queue_time_us":8416,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":186245,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":10462,"spinlock_wait_cycles":80307968,"thread_start_us":892471,"threads_started":2,"wal-append.queue_time_us":497715,"wal-append.run_cpu_time_us":3236,"wal-append.run_wall_time_us":1292081}
W20250831 07:59:38.862083 31729 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.009s sys 0.043s
W20250831 07:59:38.863847 31729 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.707s user 0.010s sys 0.043s
I20250831 07:59:38.824380 2376 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:38.778107 2277 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:38.817968 2348 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:38.822875 1966 consensus_queue.cc:1046] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20250831 07:59:38.783391 2337 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:38.856948 2262 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:38.919582 2096 consensus_meta.cc:287] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.967s user 0.008s sys 0.004s
I20250831 07:59:38.857134 2366 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:38.975958 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1447) took 2003 ms. Trace:
I20250831 07:59:38.976588 31805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 410) took 1263 ms. Trace:
I20250831 07:59:38.977725 31805 rpcz_store.cc:276] 0831 07:59:37.713666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:38.053579 (+339913us) service_pool.cc:225] Handling call
0831 07:59:38.105771 (+ 52192us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:38.783022 (+677251us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:38.976283 (+193261us) spinlock_profiling.cc:243] Waited 667 ms on lock 0x7b540015a754. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:38.976516 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1534142336}
I20250831 07:59:38.977681 31878 rpcz_store.cc:276] 0831 07:59:36.973090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:37.809090 (+836000us) service_pool.cc:225] Handling call
0831 07:59:37.809374 (+ 284us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:37.809593 (+ 219us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:37.809620 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:37.815762 (+ 6142us) log.cc:844] Serialized 26 byte log entry
0831 07:59:38.753428 (+937666us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:38.753487 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:38.753586 (+ 99us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:38.975696 (+222110us) raft_consensus.cc:1683] finished
0831 07:59:38.975717 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:38.975892 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":867791,"threads_started":1,"wal-append.queue_time_us":1051875}
I20250831 07:59:38.916900 2242 consensus_queue.cc:1046] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.060s
I20250831 07:59:38.962162 2352 raft_consensus.cc:3053] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:38.955766 2241 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:38.985749 2319 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:38.973438 2096 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:38.988807 2352 raft_consensus.cc:738] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:39.018003 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1449) took 2031 ms. Trace:
I20250831 07:59:39.023435 31882 rpcz_store.cc:276] 0831 07:59:36.986450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:37.833667 (+847217us) service_pool.cc:225] Handling call
0831 07:59:37.834090 (+ 423us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:37.887355 (+ 53265us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:37.887387 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:37.897138 (+ 9751us) log.cc:844] Serialized 26 byte log entry
0831 07:59:38.716817 (+819679us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:38.717115 (+ 298us) log.cc:844] Serialized 14 byte log entry
0831 07:59:38.773272 (+ 56157us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:38.773373 (+ 101us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:39.017717 (+244344us) raft_consensus.cc:1683] finished
0831 07:59:39.017749 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:39.017919 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":107535,"prepare.queue_time_us":198094,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":1671,"raft.queue_time_us":157916,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":131,"thread_start_us":700695,"threads_started":1,"wal-append.queue_time_us":879501}
I20250831 07:59:39.027148 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1870) took 1525 ms. Trace:
I20250831 07:59:39.027736 31802 rpcz_store.cc:276] 0831 07:59:37.502070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:37.753681 (+251611us) service_pool.cc:225] Handling call
0831 07:59:37.753986 (+ 305us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:37.891409 (+137423us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:37.891454 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:37.891512 (+ 58us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:37.891557 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:39.026818 (+1135261us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b54001d9f54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:39.026886 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:39.027066 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":315458432}
W20250831 07:59:39.007382 2301 log.cc:927] Time spent T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20250831 07:59:39.057504 2352 consensus_queue.cc:260] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.074702 2348 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:39.082929 2262 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:39.074889 2352 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:39.101781 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:39.121052 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 314693us
W20250831 07:59:39.121631 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 363732us
I20250831 07:59:39.124347 31699 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:39.124943 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 352223us
I20250831 07:59:39.127398 31775 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:39.136464 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:39.137193 31882 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:39.138266 31699 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:39.143450 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.144241 31878 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:39.100618 2277 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:39.102604 2259 raft_consensus.cc:695] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:39.145951 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:39.168912 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1437) took 3053 ms. Trace:
I20250831 07:59:39.159137 31699 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:39.148218 31883 raft_consensus.cc:1273] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:39.167416 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.169027 31878 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 8: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:39.156824 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.169441 31879 rpcz_store.cc:276] 0831 07:59:36.115228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.145211 (+ 29983us) service_pool.cc:225] Handling call
0831 07:59:36.179136 (+ 33925us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:36.179484 (+ 348us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:36.179527 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:36.179983 (+ 456us) log.cc:844] Serialized 26 byte log entry
0831 07:59:36.321746 (+141763us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:37.758993 (+1437247us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b1800111040. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce7c4955c 00007fdce507e77b 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce5098805 00007fdceb9e2327 00007fdce8007957 00007fdce801706f
0831 07:59:37.759146 (+ 153us) log.cc:844] Serialized 14 byte log entry
0831 07:59:37.759338 (+ 192us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:37.759414 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:38.869542 (+1110128us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b1800111040. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b792 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:59:39.168714 (+299172us) raft_consensus.cc:1683] finished
0831 07:59:39.168737 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:39.168854 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67894,"prepare.queue_time_us":247443,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":45211,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":15151,"spinlock_wait_cycles":30489728,"thread_start_us":311888,"threads_started":2,"wal-append.queue_time_us":475887}
I20250831 07:59:39.170997 31847 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:39.175020 31847 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:39.173835 2259 consensus_queue.cc:237] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.289439 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 1921) took 2747 ms. Trace:
I20250831 07:59:39.206231 2277 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:39.290117 31725 rpcz_store.cc:276] 0831 07:59:36.536998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.537310 (+ 312us) service_pool.cc:225] Handling call
0831 07:59:36.563079 (+ 25769us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:37.421750 (+858671us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:39.284315 (+1862565us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b540038b8d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:39.284532 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1968228608}
W20250831 07:59:39.175836 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 48469.2us
I20250831 07:59:39.180116 2096 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:39.144451 2242 raft_consensus.cc:2802] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 07:59:39.181054 31806 raft_consensus.cc:1215] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Deduplicated request from leader. Original: 4.6->[6.7-6.7] Dedup: 6.7->[]
I20250831 07:59:39.168437 2124 raft_consensus.cc:2802] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:39.182679 1378 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:39.183538 2366 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:39.201714 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.274565 2294 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:39.202978 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.308787 31881 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
W20250831 07:59:39.205785 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:39.214056 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:39.295739 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 120830us
W20250831 07:59:39.221715 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:39.312886 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 360838us
I20250831 07:59:39.271984 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 312) took 1131 ms. Trace:
I20250831 07:59:39.272768 31808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1715) took 4428 ms. Trace:
W20250831 07:59:39.269574 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 80323.5us
I20250831 07:59:39.142170 2316 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 07:59:39.275023 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.281576 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.275331 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.279625 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.282368 2135 consensus_queue.cc:1046] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20250831 07:59:39.285550 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.324724 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.325505 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.294950 31775 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:39.277201 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:39.266927 2256 raft_consensus.cc:2802] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:39.145190 2385 consensus_queue.cc:1035] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.148s
I20250831 07:59:39.302906 31806 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250831 07:59:39.307286 2124 raft_consensus.cc:695] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:39.311090 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:39.342458 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:39.343297 31805 raft_consensus.cc:2408] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250831 07:59:39.314188 31808 rpcz_store.cc:276] 0831 07:59:34.830471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:34.831123 (+ 652us) service_pool.cc:225] Handling call
0831 07:59:36.049470 (+1218347us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:36.102286 (+ 52816us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:39.258419 (+3156133us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54000ce9d0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:39.258647 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2801201536}
I20250831 07:59:39.313923 31729 rpcz_store.cc:276] 0831 07:59:38.140992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:38.141896 (+ 904us) service_pool.cc:225] Handling call
0831 07:59:38.142209 (+ 313us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:38.142452 (+ 243us) raft_consensus.cc:1532] Early marking committed up to index 7
0831 07:59:38.864328 (+721876us) log.cc:844] Serialized 14 byte log entry
0831 07:59:39.271524 (+407196us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:39.271600 (+ 76us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:39.271672 (+ 72us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:39.271763 (+ 91us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:39.271920 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":748962,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":3879,"raft.queue_time_us":33102,"thread_start_us":1114664,"threads_started":2}
I20250831 07:59:39.315008 31699 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:39.298204 2262 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:39.317047 31809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1871) took 1814 ms. Trace:
I20250831 07:59:39.270076 31882 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:39.318064 31875 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:39.323077 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.387641 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.387781 31881 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:39.388423 31882 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:39.388934 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.389460 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.389678 31881 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:39.390441 31882 raft_consensus.cc:2391] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 8: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:39.390739 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.319098 31879 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:39.300961 2242 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:39.326061 31878 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:39.331545 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:39.333036 31775 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:39.333380 2125 raft_consensus.cc:2747] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 07:59:39.429497 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 88725.4us
I20250831 07:59:39.334595 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:39.436954 31879 raft_consensus.cc:1273] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20250831 07:59:39.339622 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:39.308256 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:39.384397 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:39.385213 31809 rpcz_store.cc:276] 0831 07:59:37.502407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:37.797485 (+295078us) service_pool.cc:225] Handling call
0831 07:59:37.797779 (+ 294us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:37.798004 (+ 225us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 07:59:37.798256 (+ 252us) log.cc:844] Serialized 14 byte log entry
0831 07:59:39.316599 (+1518343us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:39.316670 (+ 71us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:39.316743 (+ 73us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:39.316838 (+ 95us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:39.316978 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50661,"prepare.queue_time_us":3650,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":51895,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":15208,"thread_start_us":1478758,"threads_started":2,"wal-append.queue_time_us":1112646}
I20250831 07:59:39.391621 31882 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8. Ops: []
I20250831 07:59:39.441280 31805 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:39.419704 31803 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:39.443535 31878 raft_consensus.cc:1273] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20250831 07:59:39.443907 31878 pending_rounds.cc:85] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10: Aborting all ops after (but not including) 3
I20250831 07:59:39.304548 2277 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.444120 31878 pending_rounds.cc:107] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250831 07:59:39.444401 31878 raft_consensus.cc:2887] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:39.447292 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:39.447152 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:39.448874 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:39.447681 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:39.451411 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:39.452037 31803 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:39.452751 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:39.447230 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:39.452803 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:39.468376 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 85770.8us
I20250831 07:59:39.469357 31803 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:39.315181 1378 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:39.471715 31875 raft_consensus.cc:1273] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20250831 07:59:39.479795 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 146826us
I20250831 07:59:39.448100 31804 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:39.473639 31773 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:39.439592 2242 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Advancing to term 9
W20250831 07:59:39.482594 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 140284us
I20250831 07:59:39.317094 2294 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.496665 2397 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:39.417482 2388 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:39.509284 31881 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:39.453395 2135 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:39.443163 2256 raft_consensus.cc:695] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:39.560067 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.449994 2391 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:39.596051 31875 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
W20250831 07:59:39.596709 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 123158us
I20250831 07:59:39.596907 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:39.597148 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 112544us
I20250831 07:59:39.428867 2262 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.598723 31775 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:39.601785 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 271571us
I20250831 07:59:39.449571 2389 raft_consensus.cc:2747] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:39.513221 2397 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:39.607528 2259 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:39.612301 2392 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:39.602679 2124 consensus_queue.cc:237] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.617056 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.696972 31875 raft_consensus.cc:2391] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:39.621188 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:39.698581 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.621322 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:39.624312 2256 consensus_queue.cc:237] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.595651 2135 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:39.626446 2296 raft_consensus.cc:2802] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 07:59:39.633139 2394 log.cc:927] Time spent T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.251s user 0.001s sys 0.000s
I20250831 07:59:39.633260 1378 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:39.634689 2277 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:39.635632 2376 consensus_meta.cc:287] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.714s user 0.008s sys 0.002s
I20250831 07:59:39.640981 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.780251 2376 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.781055 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.781678 31883 raft_consensus.cc:2466] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:39.782366 2376 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:39.782464 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:39.645623 2337 consensus_meta.cc:287] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.721s user 0.009s sys 0.002s
I20250831 07:59:39.783859 31874 raft_consensus.cc:2391] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:39.783100 31883 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8.
I20250831 07:59:39.649981 2387 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:39.682427 31880 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:39.684989 2386 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:39.602769 2319 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:39.609567 2348 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:39.603843 2391 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.687438 2330 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:39.686935 31773 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:39.733484 2366 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:39.691401 2216 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:39.693805 2349 raft_consensus.cc:3053] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:39.694939 31848 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:39.518975 2367 consensus_queue.cc:1046] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.7, Next index: 7, Last known committed idx: 6, Time since last communication: 0.092s
I20250831 07:59:39.741977 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:39.804625 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 109786us
I20250831 07:59:39.480046 2271 consensus_queue.cc:1035] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.201s
I20250831 07:59:39.752171 2389 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:39.756577 2393 log.cc:927] Time spent T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.277s user 0.000s sys 0.000s
I20250831 07:59:39.763767 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 305) took 1644 ms. Trace:
I20250831 07:59:39.791709 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.681506 2397 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:39.740868 2135 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:39.795611 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 108794us
I20250831 07:59:39.655320 2294 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:39.735759 2259 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:39.741160 2296 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
W20250831 07:59:39.789819 2352 consensus_meta.cc:287] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.689s user 0.010s sys 0.000s
W20250831 07:59:39.800853 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:39.802803 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.813680 2286 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:39.805398 31848 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:39.883320 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 78052.4us
I20250831 07:59:39.884215 31883 raft_consensus.cc:2391] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
W20250831 07:59:39.750677 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:39.846459 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 531466us
I20250831 07:59:39.868489 31733 rpcz_store.cc:276] 0831 07:59:38.119763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:38.119938 (+ 175us) service_pool.cc:225] Handling call
0831 07:59:38.120299 (+ 361us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:38.776632 (+656333us) raft_consensus.cc:1532] Early marking committed up to index 9
0831 07:59:39.097738 (+321106us) log.cc:844] Serialized 14 byte log entry
0831 07:59:39.483772 (+386034us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:39.488983 (+ 5211us) raft_consensus.cc:1643] Marking committed up to 9
0831 07:59:39.509021 (+ 20038us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:39.763434 (+254413us) spinlock_profiling.cc:243] Waited 656 ms on lock 0x7b54003e1f54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:39.763498 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:39.763695 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":71106,"prepare.queue_time_us":147135,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":2125,"raft.queue_time_us":182401,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":45772,"spinlock_wait_cycles":1508778752,"thread_start_us":633795,"threads_started":2,"wal-append.queue_time_us":399339}
I20250831 07:59:39.795147 2319 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:39.804216 2349 raft_consensus.cc:738] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:39.794624 2386 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:39.743541 2392 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.797785 2330 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:39.878845 31773 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:39.814596 2278 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:39.804713 31882 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
W20250831 07:59:39.879779 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 281131us
I20250831 07:59:39.807189 2366 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.881996 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.884995 31699 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:39.788928 2337 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.884315 31848 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:39.879801 2367 consensus_queue.cc:1230] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer 7512ba2ed3a94982a22a75f51a647e65 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250831 07:59:39.801544 2348 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:39.886986 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 318867us
I20250831 07:59:39.875309 2262 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:39.889834 31773 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:39.880177 2296 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:39.892138 31805 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20250831 07:59:39.892045 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:39.881296 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 93230.8us
I20250831 07:59:39.888499 2319 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:39.894636 31848 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:39.867403 2242 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:39.896469 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.918217 31882 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
I20250831 07:59:39.892637 2292 raft_consensus.cc:2802] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 07:59:39.899521 31699 connection.cc:232] RPC call timeout handler was delayed by -1.75542396000003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17120000000000002s for a response
I20250831 07:59:39.876144 2125 consensus_queue.cc:1035] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20250831 07:59:39.910676 31849 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:39.893517 31875 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
I20250831 07:59:39.911377 31773 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:39.922303 31733 raft_consensus.cc:1238] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20250831 07:59:39.923137 31849 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:39.923870 32464 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8. Ops: []
I20250831 07:59:39.924222 2386 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:39.912683 2256 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:39.924448 2096 consensus_queue.cc:1046] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250831 07:59:39.924860 2256 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:39.925242 2256 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:39.927167 2294 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:39.919539 2292 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:39.928237 2292 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:39.928484 2391 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:39.929512 2391 raft_consensus.cc:2747] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:39.931201 2348 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:39.927672 2294 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.933207 2392 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:39.933885 2294 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:39.874830 2389 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:39.893365 31775 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:39.796702 2387 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:39.888942 2277 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:39.948737 2277 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:39.948879 2389 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:39.949142 2277 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:39.950631 2360 log.cc:927] Time spent T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 1.349s user 0.014s sys 0.025s
I20250831 07:59:39.916371 31848 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:39.951952 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 300) took 3349 ms. Trace:
I20250831 07:59:39.914881 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:39.918713 31809 raft_consensus.cc:2391] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:39.952946 2349 consensus_queue.cc:260] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:39.954344 2349 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:39.956699 2389 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:39.890601 2278 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:39.919054 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.882772 2286 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:39.896859 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:39.892284 2157 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:39.909744 2352 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:39.908715 31879 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:39.923179 1378 raft_consensus.cc:2802] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:39.940667 2136 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:39.942066 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1506) took 1604 ms. Trace:
I20250831 07:59:39.942272 2400 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:39.943639 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:40.204914 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.206103 31734 raft_consensus.cc:2408] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250831 07:59:40.017283 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.204631 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:40.204303 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 292949us
I20250831 07:59:40.210367 31773 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:39.952332 31727 rpcz_store.cc:276] 0831 07:59:36.602397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.602565 (+ 168us) service_pool.cc:225] Handling call
0831 07:59:36.602960 (+ 395us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:37.148952 (+545992us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:37.148973 (+ 21us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:37.149355 (+ 382us) log.cc:844] Serialized 26 byte log entry
0831 07:59:38.108328 (+958973us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:38.108366 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:39.560434 (+1452068us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54001aa754. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:39.560510 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:39.876537 (+316027us) spinlock_profiling.cc:243] Waited 78.1 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:39.951698 (+ 75161us) raft_consensus.cc:1683] finished
0831 07:59:39.951721 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:39.951891 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53334,"raft.queue_time_us":67163,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":9397,"spinlock_wait_cycles":505954816,"thread_start_us":1142011,"threads_started":2,"wal-append.queue_time_us":1199451}
I20250831 07:59:40.210741 32434 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:40.211987 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.017596 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.077800 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:40.212153 31732 rpcz_store.cc:276] 0831 07:59:38.337614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:38.337938 (+ 324us) service_pool.cc:225] Handling call
0831 07:59:38.338299 (+ 361us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 07:59:38.340698 (+ 2399us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:38.340741 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 07:59:38.713585 (+372844us) log.cc:844] Serialized 52 byte log entry
0831 07:59:39.154117 (+440532us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:39.154176 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:39.154269 (+ 93us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:39.941860 (+787591us) raft_consensus.cc:1683] finished
0831 07:59:39.941879 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:39.942003 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":142345,"raft.queue_time_us":2948,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":16073,"thread_start_us":365832,"threads_started":1,"wal-append.queue_time_us":548976}
I20250831 07:59:40.213866 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 416) took 1286 ms. Trace:
W20250831 07:59:40.214174 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 291093us
I20250831 07:59:40.217618 31801 rpcz_store.cc:276] 0831 07:59:38.928045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:38.928213 (+ 168us) service_pool.cc:225] Handling call
0831 07:59:38.928628 (+ 415us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:38.928990 (+ 362us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:38.929025 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:39.686944 (+757919us) spinlock_profiling.cc:243] Waited 7.9 ms on lock 0x7b2c00000c88. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce69a09a1 00007fdce69affd2 00007fdce8039021 00007fdce7ffd3fa 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32
0831 07:59:39.687385 (+ 441us) log.cc:844] Serialized 26 byte log entry
0831 07:59:39.884884 (+197499us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:39.884920 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:39.884979 (+ 59us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:39.964268 (+ 79289us) raft_consensus.cc:1683] finished
0831 07:59:39.964302 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:40.213807 (+249505us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":439,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":20141,"spinlock_wait_cycles":18167040,"thread_start_us":197098,"threads_started":1,"wal-append.queue_time_us":227425}
W20250831 07:59:40.108182 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:40.213912 2388 consensus_queue.cc:1046] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20250831 07:59:40.224110 2388 raft_consensus.cc:3053] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 07:59:40.224468 2388 raft_consensus.cc:738] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:40.225248 2388 consensus_queue.cc:260] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:40.226508 2388 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:39.960433 2242 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:40.073486 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:40.228449 2157 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.214933 31800 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:40.230537 2157 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:40.231122 2350 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:40.231441 2366 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:40.231676 2350 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:40.232197 31800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1728) took 3262 ms (client timeout 1712 ms). Trace:
I20250831 07:59:40.189677 2409 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250831 07:59:40.232796 2395 log.cc:927] Time spent T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.665s user 0.007s sys 0.003s
I20250831 07:59:40.232911 2409 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:40.233507 2136 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.235477 2136 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:40.214762 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.250995 31730 raft_consensus.cc:2408] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20250831 07:59:40.252961 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 311084us
I20250831 07:59:40.016016 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:40.231660 2092 raft_consensus.cc:3058] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Advancing to term 9
W20250831 07:59:40.236944 2406 consensus_queue.cc:1162] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a0d845d01c8c4894895f66e3beae298c" responder_term: 9 status { last_received { term: 6 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 8. Current term is 9. Ops: [8.5-8.5]" } } last_received_current_leader { term: 0 index: 0 } }
I20250831 07:59:40.223091 2374 raft_consensus.cc:2802] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:40.237391 2277 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.264927 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:40.204959 2296 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.238998 2135 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.239722 2099 raft_consensus.cc:2802] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:40.240374 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39068 (request call id 728) took 1091 ms. Trace:
I20250831 07:59:40.096578 2392 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:36.323887 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:59:40.233527 2409 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:40.211690 2286 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:40.211553 2352 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:40.246982 2341 raft_consensus.cc:3053] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Stepping down as leader of term 7
I20250831 07:59:40.248252 2330 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:40.248289 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:40.379398 31883 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:39.956923 2259 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:39.920189 2262 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:40.262868 2374 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:59:40.232661 31800 rpcz_store.cc:269] 0831 07:59:36.969667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:37.353250 (+383583us) service_pool.cc:225] Handling call
0831 07:59:40.231957 (+2878707us) spinlock_profiling.cc:243] Waited 2.86 s on lock 0x7b540003a2d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:40.232069 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6574078848}
W20250831 07:59:40.095678 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 202353us
I20250831 07:59:40.188534 2256 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.443468 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1463) took 1236 ms. Trace:
I20250831 07:59:40.449791 31874 rpcz_store.cc:276] 0831 07:59:39.206912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:39.207802 (+ 890us) service_pool.cc:225] Handling call
0831 07:59:40.443265 (+1235463us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b540020dad4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:40.443409 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1069630848}
I20250831 07:59:40.255352 31729 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:40.214694 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.260373 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.204425 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.261721 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.260404 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.244864 2204 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:40.265571 31875 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:39.935091 2405 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:39.958684 2337 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:40.241611 2397 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.267001 31802 rpcz_store.cc:276] 0831 07:59:39.149124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:39.149332 (+ 208us) service_pool.cc:225] Handling call
0831 07:59:40.239844 (+1090512us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54002f0654. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f81cd0 00007fdce8005eea 00007fdcecf5e60e 00007fdce65fbcc7 00007fdce65fbc52 00007fdce65fbb81 00007fdce65fbafc 00007fdce65fadb2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:40.240316 (+ 472us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":404701568}
I20250831 07:59:40.216152 2389 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:40.269757 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:40.255970 2278 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:40.160720 2410 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:40.253831 1378 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:40.334422 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:40.414906 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:40.416416 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:40.215710 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.427484 31728 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:40.432974 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:40.268806 2392 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:40.437187 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:40.437327 2387 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:40.263674 2400 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.440099 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:40.441743 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:40.441941 2310 consensus_queue.cc:1035] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20250831 07:59:40.443298 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:40.268532 2368 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:40.268761 2099 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:40.269966 2136 consensus_queue.cc:1046] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.174s
I20250831 07:59:40.444097 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:40.444914 31803 raft_consensus.cc:1238] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:40.433441 2366 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:40.439375 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:40.608780 31698 leader_election.cc:336] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.555s (SENT)
I20250831 07:59:40.529083 31731 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:40.623865 31727 raft_consensus.cc:2408] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250831 07:59:40.527845 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:40.638386 31730 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
W20250831 07:59:40.452950 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 241700us
I20250831 07:59:40.625628 31732 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:40.541834 31883 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:40.539438 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:40.640082 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:40.596526 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 223291us
I20250831 07:59:40.639899 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1845) took 1017 ms. Trace:
I20250831 07:59:40.641222 31734 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:40.641597 31883 rpcz_store.cc:276] 0831 07:59:39.622093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:40.211939 (+589846us) service_pool.cc:225] Handling call
0831 07:59:40.639844 (+427905us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:40.642102 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.519032 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:40.643357 31804 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:40.642943 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:40.640290 31874 raft_consensus.cc:2374] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4. Re-sending same reply.
I20250831 07:59:40.454598 31775 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:40.636271 31726 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:40.639258 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.641085 31732 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:40.388859 2341 raft_consensus.cc:738] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:40.548815 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1471) took 1227 ms. Trace:
I20250831 07:59:40.571722 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:40.647070 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.648106 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:40.704222 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 57938.6us
I20250831 07:59:40.716130 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.716737 31732 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:40.716837 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:40.711903 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1846) took 1089 ms. Trace:
I20250831 07:59:40.678368 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:40.648900 1378 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:40.538914 2292 consensus_meta.cc:287] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.610s user 0.007s sys 0.003s
W20250831 07:59:40.693711 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 239214us
I20250831 07:59:40.561650 2135 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:40.567718 2296 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:40.729490 31727 raft_consensus.cc:1238] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:40.730966 31728 raft_consensus.cc:1238] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 10 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 9. Current term is 10. Ops: []
I20250831 07:59:40.686619 2366 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.733696 2366 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:40.733990 2292 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:40.735888 2292 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:40.715598 31729 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 9.
I20250831 07:59:40.741205 31806 raft_consensus.cc:1215] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Deduplicated request from leader. Original: 4.6->[6.7-6.7] Dedup: 6.7->[]
I20250831 07:59:40.741729 31806 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250831 07:59:40.695058 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:40.448052 2374 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:40.747746 31876 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:40.646571 2410 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:40.397114 2352 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:40.698450 31805 raft_consensus.cc:1238] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:40.752040 2410 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:40.432101 2262 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:40.752844 2262 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:40.754010 2422 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:40.754545 2422 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.757077 2350 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:40.389233 2414 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:40.757937 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1837) took 1189 ms. Trace:
I20250831 07:59:40.757742 2414 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.758701 2397 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:40.698033 2419 consensus_queue.cc:1035] T 3054807582f54efba5d989f569ab0666 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.102s
I20250831 07:59:40.759874 2414 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:40.623843 2099 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:40.760972 2262 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.685972 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:40.761910 31875 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
W20250831 07:59:40.762302 2349 consensus_meta.cc:287] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.807s user 0.007s sys 0.003s
I20250831 07:59:40.762791 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:40.763403 31875 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:40.698777 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:40.562901 2361 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:40.764262 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:40.708817 31730 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:40.501477 2385 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:40.651243 2389 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:40.500985 2259 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:40.650704 2392 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:40.700078 31773 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:40.618742 2319 consensus_meta.cc:287] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.676s user 0.010s sys 0.001s
I20250831 07:59:40.710616 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.789928 31734 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:40.716647 2387 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:40.718313 31879 rpcz_store.cc:276] 0831 07:59:39.622402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:40.571437 (+949035us) service_pool.cc:225] Handling call
0831 07:59:40.711844 (+140407us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:40.806936 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:40.712759 31802 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 8.
I20250831 07:59:40.719431 31775 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:40.698201 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 46137.3us
I20250831 07:59:40.711458 31881 rpcz_store.cc:276] 0831 07:59:39.320932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:39.390485 (+ 69553us) service_pool.cc:225] Handling call
0831 07:59:40.548614 (+1158129us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b5400410d54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:40.548745 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":270990720}
I20250831 07:59:40.645186 2368 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:40.491423 2363 consensus_queue.cc:1035] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.219s
W20250831 07:59:40.823338 31847 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:57346 dropped due to backpressure: service queue is full with 50 items [suppressed 4 similar messages]
I20250831 07:59:40.651027 2270 consensus_queue.cc:1046] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20250831 07:59:40.722836 2400 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:40.724056 2391 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:40.725840 2186 consensus_queue.cc:260] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:40.654534 2405 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:40.651124 32464 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8.
I20250831 07:59:40.704672 1804 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250831 07:59:40.650125 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:40.687182 2403 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:40.709396 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:40.769313 31698 leader_election.cc:336] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.691s (SENT)
I20250831 07:59:40.758317 31878 rpcz_store.cc:276] 0831 07:59:39.568934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:39.569068 (+ 134us) service_pool.cc:225] Handling call
0831 07:59:40.549004 (+979936us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:40.549279 (+ 275us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:40.549316 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:40.549377 (+ 61us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:40.549423 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:40.549498 (+ 75us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:40.757771 (+208273us) spinlock_profiling.cc:243] Waited 979 ms on lock 0x7b5400410d50. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:40.757881 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2252758912}
I20250831 07:59:40.773172 31804 raft_consensus.cc:2466] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
W20250831 07:59:40.895248 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 125988us
W20250831 07:59:40.896049 31698 leader_election.cc:336] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.702s (SENT)
I20250831 07:59:40.896555 31698 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:40.559926 2277 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:40.798527 2409 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.791352 2319 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.807693 31879 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:40.453233 2370 consensus_queue.cc:1035] T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.269s
I20250831 07:59:40.788403 2148 raft_consensus.cc:2747] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:40.825078 31882 raft_consensus.cc:2366] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:40.822925 1947 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250831 07:59:40.947127 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 51114.8us
I20250831 07:59:40.799255 2278 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:40.844394 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:40.764410 2361 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:40.895136 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1479) took 1202 ms. Trace:
I20250831 07:59:40.960453 31883 rpcz_store.cc:276] 0831 07:59:39.692379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:40.672123 (+979744us) service_pool.cc:225] Handling call
0831 07:59:40.895083 (+222960us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:40.960772 31848 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:40.776868 2259 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:40.920315 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:40.961848 32434 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:40.940521 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:40.963004 31802 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
W20250831 07:59:40.941864 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:40.948045 31698 leader_election.cc:336] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.712s (SENT)
W20250831 07:59:40.969223 31698 leader_election.cc:336] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.730s (SENT)
W20250831 07:59:40.946736 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 246756us
I20250831 07:59:40.971305 31773 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:40.973312 31773 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:40.974743 31773 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:40.951629 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 232247us
I20250831 07:59:40.953238 2368 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:40.953670 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 248479us
I20250831 07:59:40.823711 2387 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.892927 1804 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:40.777458 2385 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:40.993542 2374 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:40.995004 2385 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:40.995122 32470 raft_consensus.cc:1238] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
W20250831 07:59:40.845472 2096 consensus_queue.cc:1162] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7512ba2ed3a94982a22a75f51a647e65" responder_term: 7 status { last_received { term: 7 index: 8 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: [6.7-6.7]" } } last_received_current_leader { term: 0 index: 0 } }
I20250831 07:59:40.958997 31808 raft_consensus.cc:2391] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:40.996829 2262 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:40.998636 2259 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:40.946967 1947 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:40.999364 1947 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:41.000125 31732 raft_consensus.cc:1238] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 8. Current term is 9. Ops: []
I20250831 07:59:41.000485 2389 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.001650 31875 raft_consensus.cc:2366] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:40.999305 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.003124 31728 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:40.959250 1378 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.006096 31874 raft_consensus.cc:2366] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5.
I20250831 07:59:41.006290 2387 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:40.670994 2092 consensus_queue.cc:1046] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.101s
I20250831 07:59:40.913151 2251 raft_consensus.cc:2747] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:40.977573 31803 raft_consensus.cc:2391] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:59:41.013208 2286 consensus_meta.cc:287] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.585s user 0.004s sys 0.004s
I20250831 07:59:41.013573 2389 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:40.984187 32435 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:40.986670 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.959707 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:40.892905 2256 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:40.891582 2391 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.014498 2374 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.018599 2391 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:41.007285 2204 raft_consensus.cc:2802] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:41.019466 2204 raft_consensus.cc:695] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:41.020402 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:40.827586 2352 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.021500 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.022663 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.023686 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.022276 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:40.984328 2272 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:41.026590 2272 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:41.026989 2272 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:40.913631 2403 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:41.010371 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.017060 1804 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:41.031261 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.032608 31805 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20250831 07:59:41.011500 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:40.984606 2279 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:41.035646 32469 raft_consensus.cc:1273] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20250831 07:59:41.036177 32469 pending_rounds.cc:85] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: Aborting all ops after (but not including) 7
I20250831 07:59:41.036402 32469 pending_rounds.cc:107] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20250831 07:59:41.036653 32469 raft_consensus.cc:2887] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:40.979274 31773 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:41.015650 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:41.038789 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:40.985090 2295 raft_consensus.cc:2747] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:41.015439 2319 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.041833 2204 consensus_queue.cc:237] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.043385 2422 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:41.023618 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.024556 31730 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:41.009047 2405 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:40.985764 2148 raft_consensus.cc:2802] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:41.006640 1378 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:41.016269 2348 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:41.016999 2409 leader_election.cc:290] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.048276 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:41.014331 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.035866 2286 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:40.982905 2277 consensus_queue.cc:1046] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20250831 07:59:41.060247 31805 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:41.063056 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.055285 2370 raft_consensus.cc:2802] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 07:59:41.055400 2148 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:41.052999 2391 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:41.056603 2348 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:41.082123 2370 raft_consensus.cc:695] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:41.086820 31876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1448) took 4104 ms (client timeout 2151 ms). Trace:
W20250831 07:59:41.088028 31876 rpcz_store.cc:269] 0831 07:59:36.978412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.995345 (+ 16933us) service_pool.cc:225] Handling call
0831 07:59:41.082857 (+4087512us) spinlock_profiling.cc:243] Waited 3.64 s on lock 0x7b54000f8a54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:41.082955 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8374896896}
I20250831 07:59:41.062367 2228 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:41.085450 2148 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:41.086542 2348 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:41.089591 2391 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:41.156664 2278 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.156605 2370 consensus_queue.cc:237] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:41.153119 2334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.319s user 0.038s sys 0.029s
I20250831 07:59:41.162256 2352 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:41.155244 2341 consensus_queue.cc:260] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:41.164162 2392 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.162398 2403 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.165136 2391 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:41.164727 2341 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Advancing to term 8
W20250831 07:59:41.198184 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 227180us
I20250831 07:59:41.254474 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.255379 31698 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:41.260426 2405 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 07:59:41.260568 2408 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:41.261044 2405 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:41.261581 2405 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:41.261494 2408 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:41.199959 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.025102 31733 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:41.268141 2302 consensus_queue.cc:1046] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250831 07:59:41.270082 2302 raft_consensus.cc:3053] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 07:59:41.270545 2302 raft_consensus.cc:738] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:41.204293 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:41.053696 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 74471.5us
W20250831 07:59:41.272895 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.273135 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.273504 31773 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:41.209782 31730 raft_consensus.cc:1273] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:41.274188 31729 raft_consensus.cc:2408] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250831 07:59:41.274945 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.275138 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:41.275970 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 158587us
I20250831 07:59:41.276436 2350 raft_consensus.cc:2802] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:41.277038 2350 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:41.277529 2350 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:41.278210 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:41.279225 2391 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:41.166851 2392 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.170652 2410 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:41.280095 2391 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:41.283073 2391 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:41.283145 2410 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:41.232368 2217 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:41.284386 2217 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:41.275552 31773 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:41.287736 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.288244 31801 raft_consensus.cc:1238] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:41.288548 1966 raft_consensus.cc:3053] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 3 LEADER]: Stepping down as leader of term 3
I20250831 07:59:41.289113 1966 raft_consensus.cc:738] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:41.290556 31775 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:41.290951 2392 raft_consensus.cc:2747] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:41.292155 2392 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:41.292711 2392 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:41.293191 2392 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:41.293553 2408 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.233660 2286 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:40.823757 31847 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:57346 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1864)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1868)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 319)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 314)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1857)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1879)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1861)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1874)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 316)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1859)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1885)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1881)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1855)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1850)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1886)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1887)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1884)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1877)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1842)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 306)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 308)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 309)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 310)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 311)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1475)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1476)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1854)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 315)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 317)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 318)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1483)
I20250831 07:59:41.132889 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.302753 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.303627 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1866) took 1074 ms. Trace:
I20250831 07:59:41.304157 31876 rpcz_store.cc:276] 0831 07:59:40.229065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.132593 (+903528us) service_pool.cc:225] Handling call
0831 07:59:41.303389 (+170796us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.305884 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:41.306717 31876 raft_consensus.cc:2374] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5. Re-sending same reply.
I20250831 07:59:41.307283 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1868) took 1072 ms. Trace:
I20250831 07:59:41.250125 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:41.250852 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.275216 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:41.159598 2334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.328s user 0.038s sys 0.030s
I20250831 07:59:41.298712 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.300124 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.319700 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.307650 31876 rpcz_store.cc:276] 0831 07:59:40.234614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.305329 (+1070715us) service_pool.cc:225] Handling call
0831 07:59:41.307195 (+ 1866us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.333981 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.290627 1966 consensus_queue.cc:260] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:41.341675 31809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1890) took 1987 ms (client timeout 1994 ms). Trace:
W20250831 07:59:41.345211 31882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1856) took 1430 ms (client timeout 1522 ms). Trace:
I20250831 07:59:41.347309 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:41.351532 1378 raft_consensus.cc:3053] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:41.391536 31730 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:41.391808 1378 raft_consensus.cc:738] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:41.392438 2220 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:41.392503 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.392933 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.393211 1378 consensus_queue.cc:260] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.393828 31730 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:41.394037 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.287729 2217 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:59:41.395890 31806 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:41.397691 31806 raft_consensus.cc:1238] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:41.391623 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1864) took 1163 ms. Trace:
I20250831 07:59:41.400436 31883 rpcz_store.cc:276] 0831 07:59:40.228310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.346569 (+1118259us) service_pool.cc:225] Handling call
0831 07:59:41.391526 (+ 44957us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.401515 31878 raft_consensus.cc:2391] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:41.402016 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1867) took 1167 ms. Trace:
I20250831 07:59:41.402397 31878 rpcz_store.cc:276] 0831 07:59:40.234285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.329142 (+1094857us) service_pool.cc:225] Handling call
0831 07:59:41.401936 (+ 72794us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.401830 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.300971 2302 consensus_queue.cc:260] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:41.394907 31732 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:41.426558 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:41.390723 31882 rpcz_store.cc:269] 0831 07:59:39.914229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:39.918820 (+ 4591us) service_pool.cc:225] Handling call
0831 07:59:41.344583 (+1425763us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b5400247554. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:41.344945 (+ 362us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1409389824}
I20250831 07:59:41.430019 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:41.431022 31728 raft_consensus.cc:3058] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 7
W20250831 07:59:41.431954 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.404224 1378 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:41.404582 31809 rpcz_store.cc:269] 0831 07:59:39.338160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:39.446822 (+108662us) service_pool.cc:225] Handling call
0831 07:59:41.319990 (+1873168us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b5400301954. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:41.321483 (+ 1493us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1081684864}
I20250831 07:59:41.433205 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.434295 31876 raft_consensus.cc:2366] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:41.434788 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1871) took 1199 ms. Trace:
I20250831 07:59:41.434965 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:41.435243 31876 rpcz_store.cc:276] 0831 07:59:40.235467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.432811 (+1197344us) service_pool.cc:225] Handling call
0831 07:59:41.434703 (+ 1892us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:41.456730 31846 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.403627 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.459141 31878 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:41.430054 1966 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
W20250831 07:59:41.462641 31874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1863) took 1234 ms (client timeout 1555 ms). Trace:
W20250831 07:59:41.463260 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 110400us
W20250831 07:59:41.463915 31880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1461) took 2259 ms (client timeout 2662 ms). Trace:
I20250831 07:59:41.464226 31883 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 4.
W20250831 07:59:41.464391 31880 rpcz_store.cc:269] 0831 07:59:39.204058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:39.204198 (+ 140us) service_pool.cc:225] Handling call
0831 07:59:41.463501 (+2259303us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b540007d5d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:41.463673 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":838118144}
I20250831 07:59:41.464933 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 319) took 1124 ms. Trace:
I20250831 07:59:41.465303 31883 rpcz_store.cc:276] 0831 07:59:40.340541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.401350 (+1060809us) service_pool.cc:225] Handling call
0831 07:59:41.464875 (+ 63525us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.466943 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.468035 31880 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
W20250831 07:59:41.456743 31849 leader_election.cc:336] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.521s (SENT)
I20250831 07:59:41.471616 2361 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:41.468688 31880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 314) took 1129 ms. Trace:
I20250831 07:59:41.474267 31808 raft_consensus.cc:1238] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:41.459682 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1862) took 1232 ms. Trace:
I20250831 07:59:41.475623 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:41.476061 31878 rpcz_store.cc:276] 0831 07:59:40.227706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.403161 (+1175455us) service_pool.cc:225] Handling call
0831 07:59:41.459628 (+ 56467us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.476524 31732 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:41.473363 31880 rpcz_store.cc:276] 0831 07:59:40.339345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.466474 (+1127129us) service_pool.cc:225] Handling call
0831 07:59:41.468592 (+ 2118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.477138 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.478158 31878 raft_consensus.cc:2366] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8.
I20250831 07:59:41.478655 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1857) took 1373 ms. Trace:
I20250831 07:59:41.479028 31878 rpcz_store.cc:276] 0831 07:59:40.105757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.476827 (+1371070us) service_pool.cc:225] Handling call
0831 07:59:41.478601 (+ 1774us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.480410 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1869) took 1245 ms. Trace:
I20250831 07:59:41.480829 31881 rpcz_store.cc:276] 0831 07:59:40.234878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.434422 (+1199544us) service_pool.cc:225] Handling call
0831 07:59:41.480346 (+ 45924us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:41.463090 31874 rpcz_store.cc:269] 0831 07:59:40.228009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:40.650293 (+422284us) service_pool.cc:225] Handling call
0831 07:59:41.462248 (+811955us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b54003ef154. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:41.462460 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":556133632}
I20250831 07:59:41.482209 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.478013 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.483960 31728 raft_consensus.cc:2466] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:41.394914 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.489377 31729 raft_consensus.cc:3058] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 8
I20250831 07:59:41.489746 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.490562 31874 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:41.495450 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.496659 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:41.492868 31875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1852) took 1724 ms (client timeout 2111 ms). Trace:
W20250831 07:59:41.499969 31875 rpcz_store.cc:269] 0831 07:59:39.768608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:40.764019 (+995411us) service_pool.cc:225] Handling call
0831 07:59:41.492431 (+728412us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b54002bcd54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:41.492686 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":544326400}
I20250831 07:59:41.500413 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.500952 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:41.501302 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.502429 31875 raft_consensus.cc:2466] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:41.503062 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1870) took 1267 ms. Trace:
I20250831 07:59:41.503203 32463 raft_consensus.cc:1238] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 8. Current term is 9. Ops: []
I20250831 07:59:41.501536 31883 raft_consensus.cc:2366] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:41.504215 31875 rpcz_store.cc:276] 0831 07:59:40.235174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.500930 (+1265756us) service_pool.cc:225] Handling call
0831 07:59:41.503013 (+ 2083us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.505357 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.505712 2186 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:41.507673 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.519802 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 316) took 1180 ms. Trace:
I20250831 07:59:41.535249 31726 raft_consensus.cc:2408] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20250831 07:59:41.535661 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.536862 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1872) took 1301 ms. Trace:
I20250831 07:59:41.537089 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.537390 31882 rpcz_store.cc:276] 0831 07:59:40.235737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.527593 (+1291856us) service_pool.cc:225] Handling call
0831 07:59:41.536749 (+ 9156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.538043 31726 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:41.539088 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:41.532258 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.525274 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.541312 31878 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:41.542011 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1861) took 1314 ms. Trace:
I20250831 07:59:41.542408 31849 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:41.542459 31878 rpcz_store.cc:276] 0831 07:59:40.227427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.496414 (+1268987us) service_pool.cc:225] Handling call
0831 07:59:41.541935 (+ 45521us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.532188 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 434) took 1118 ms. Trace:
I20250831 07:59:41.545044 31801 rpcz_store.cc:276] 0831 07:59:40.413686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:40.413839 (+ 153us) service_pool.cc:225] Handling call
0831 07:59:40.414171 (+ 332us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.288811 (+874640us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.508861 (+220050us) spinlock_profiling.cc:243] Waited 874 ms on lock 0x7b54000398d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:41.532060 (+ 23199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2009056640}
I20250831 07:59:41.534924 31875 rpcz_store.cc:276] 0831 07:59:40.339722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.504994 (+1165272us) service_pool.cc:225] Handling call
0831 07:59:41.519702 (+ 14708us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.543743 2201 raft_consensus.cc:2747] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:41.550608 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.516497 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:41.545630 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.493782 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:41.541327 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:41.539974 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.544623 31849 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:41.539808 31726 raft_consensus.cc:3058] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:41.549372 31729 raft_consensus.cc:2466] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 8.
I20250831 07:59:41.553161 31880 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:41.558321 31874 raft_consensus.cc:2466] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
W20250831 07:59:41.555275 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 57211us
I20250831 07:59:41.561002 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.561029 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.563133 31875 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:41.563787 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1855) took 1678 ms. Trace:
I20250831 07:59:41.564152 31875 rpcz_store.cc:276] 0831 07:59:39.886109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.550251 (+1664142us) service_pool.cc:225] Handling call
0831 07:59:41.563723 (+ 13472us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.564262 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.567868 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:41.568461 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:41.494123 1947 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.572009 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 417) took 4925 ms. Trace:
I20250831 07:59:41.572441 32469 rpcz_store.cc:276] 0831 07:59:36.570703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.570898 (+ 195us) service_pool.cc:225] Handling call
0831 07:59:36.572500 (+ 1602us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.037378 (+4464878us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.495429 (+458051us) spinlock_profiling.cc:243] Waited 4.46 s on lock 0x7b5400173254. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:41.495714 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10263431424}
I20250831 07:59:41.574980 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.574761 2405 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:41.575574 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43230 (request call id 317) took 1559 ms. Trace:
I20250831 07:59:41.576103 32470 raft_consensus.cc:1238] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8. Ops: []
I20250831 07:59:41.575946 31808 raft_consensus.cc:2391] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:41.580132 31807 raft_consensus.cc:2466] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:41.580469 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 1933) took 1563 ms. Trace:
I20250831 07:59:41.580864 31731 rpcz_store.cc:276] 0831 07:59:40.017189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:40.017316 (+ 127us) service_pool.cc:225] Handling call
0831 07:59:41.580177 (+1562861us) spinlock_profiling.cc:243] Waited 316 ms on lock 0x7b54002ec2d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:41.580297 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":726666112}
W20250831 07:59:41.581687 31807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1725) took 5279 ms (client timeout 1555 ms). Trace:
I20250831 07:59:41.576017 31725 rpcz_store.cc:276] 0831 07:59:40.015895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:40.016047 (+ 152us) service_pool.cc:225] Handling call
0831 07:59:40.146311 (+130264us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:40.898271 (+751960us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:40.898317 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:40.898382 (+ 65us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:40.898435 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.569511 (+671076us) spinlock_profiling.cc:243] Waited 752 ms on lock 0x7b540018fe54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:41.575145 (+ 5634us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:41.575379 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1728860416}
I20250831 07:59:41.584543 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:41.582137 31807 rpcz_store.cc:269] 0831 07:59:36.302054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:36.302204 (+ 150us) service_pool.cc:225] Handling call
0831 07:59:41.581441 (+5279237us) spinlock_profiling.cc:243] Waited 5.28 s on lock 0x7b5400300554. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:41.581603 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12135387392}
I20250831 07:59:41.585562 31883 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:41.584487 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.587714 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1859) took 1361 ms. Trace:
I20250831 07:59:41.588054 31883 rpcz_store.cc:276] 0831 07:59:40.226765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.552743 (+1325978us) service_pool.cc:225] Handling call
0831 07:59:41.587658 (+ 34915us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.588997 31731 raft_consensus.cc:2391] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:41.589496 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:41.586465 2236 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:41.591387 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.591742 2236 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.595424 31728 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 8.
I20250831 07:59:41.596539 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.598047 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:41.598223 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.595081 31881 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:41.599601 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1874) took 1363 ms. Trace:
I20250831 07:59:41.599924 31881 rpcz_store.cc:276] 0831 07:59:40.250239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.496332 (+1246093us) service_pool.cc:225] Handling call
0831 07:59:41.599543 (+103211us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.593304 2201 raft_consensus.cc:2802] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:41.600600 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.602823 31730 raft_consensus.cc:2466] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 9.
I20250831 07:59:41.602773 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.605340 31876 raft_consensus.cc:2374] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7. Re-sending same reply.
I20250831 07:59:41.605795 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1873) took 1369 ms. Trace:
I20250831 07:59:41.605985 31876 rpcz_store.cc:276] 0831 07:59:40.236075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.518118 (+1282043us) service_pool.cc:225] Handling call
0831 07:59:41.605753 (+ 87635us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.606843 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:41.607334 31733 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:41.610222 31878 raft_consensus.cc:2466] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8.
I20250831 07:59:41.609061 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:41.617019 2272 consensus_meta.cc:287] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.589s user 0.015s sys 0.015s
I20250831 07:59:41.617728 2272 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:41.621634 32470 raft_consensus.cc:1238] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:41.622334 2272 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.623344 2286 consensus_queue.cc:1046] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20250831 07:59:41.624058 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.625029 2286 raft_consensus.cc:3053] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:41.625444 2286 raft_consensus.cc:738] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:41.629765 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.630460 31883 raft_consensus.cc:2366] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:41.631067 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:41.636353 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.633272 31882 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:41.637483 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:41.638175 31876 raft_consensus.cc:2366] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:41.638541 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.639050 31883 raft_consensus.cc:2366] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8.
I20250831 07:59:41.643394 31732 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:41.644415 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:41.647945 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 45605.5us
I20250831 07:59:41.650885 31880 raft_consensus.cc:2408] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250831 07:59:41.611356 1947 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.655208 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:41.653952 2332 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:41.657407 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.658138 31880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1850) took 2034 ms. Trace:
I20250831 07:59:41.658521 31880 rpcz_store.cc:276] 0831 07:59:39.623612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.569229 (+1945617us) service_pool.cc:225] Handling call
0831 07:59:41.658080 (+ 88851us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.628894 2286 consensus_queue.cc:260] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.663496 31878 raft_consensus.cc:2408] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 07:59:41.663570 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.661911 31729 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 3 FOLLOWER]: Advancing to term 5
I20250831 07:59:41.674914 31881 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:41.676388 31807 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:41.678426 31875 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:41.695394 31809 raft_consensus.cc:1273] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250831 07:59:41.672734 2099 consensus_meta.cc:287] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.825s user 0.012s sys 0.012s
I20250831 07:59:41.707839 31801 raft_consensus.cc:1273] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:41.710440 31883 raft_consensus.cc:1238] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:41.703378 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:41.719946 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:41.719165 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1842) took 2080 ms. Trace:
I20250831 07:59:41.693516 2201 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:41.727079 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1847) took 2096 ms. Trace:
I20250831 07:59:41.723057 31882 rpcz_store.cc:276] 0831 07:59:39.621278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.647867 (+2026589us) service_pool.cc:225] Handling call
0831 07:59:41.675478 (+ 27611us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.696589 (+ 21111us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:41.696630 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:41.701634 (+ 5004us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:41.701687 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.701771 (+ 84us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:41.701907 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.729962 31883 rpcz_store.cc:276] 0831 07:59:39.622726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.650735 (+2028009us) service_pool.cc:225] Handling call
0831 07:59:41.675087 (+ 24352us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.718829 (+ 43742us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.719068 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.722388 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 306) took 2029 ms. Trace:
I20250831 07:59:41.731081 31876 rpcz_store.cc:276] 0831 07:59:39.680916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.650027 (+1969111us) service_pool.cc:225] Handling call
0831 07:59:41.678137 (+ 28110us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.701070 (+ 22933us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:41.701112 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:41.709939 (+ 8827us) raft_consensus.cc:1643] Marking committed up to 8
0831 07:59:41.710008 (+ 69us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.710093 (+ 85us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:41.710227 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.662827 2332 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:41.736142 31880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 308) took 2053 ms. Trace:
I20250831 07:59:41.736609 31880 rpcz_store.cc:276] 0831 07:59:39.682387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.665234 (+1982847us) service_pool.cc:225] Handling call
0831 07:59:41.692744 (+ 27510us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.735765 (+ 43021us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:41.735787 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:41.735855 (+ 68us) raft_consensus.cc:1643] Marking committed up to 8
0831 07:59:41.735906 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.735976 (+ 70us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:41.736086 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.736631 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 309) took 2054 ms. Trace:
W20250831 07:59:41.732654 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 233602us
I20250831 07:59:41.738166 31878 rpcz_store.cc:276] 0831 07:59:39.682718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.669612 (+1986894us) service_pool.cc:225] Handling call
0831 07:59:41.716766 (+ 47154us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.736235 (+ 19469us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:41.736275 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:41.736342 (+ 67us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:41.736385 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.736455 (+ 70us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:41.736569 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:41.747180 2348 consensus_meta.cc:287] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 0.549s user 0.000s sys 0.026s
I20250831 07:59:41.747308 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:41.732299 2201 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:41.748834 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.745998 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:41.751681 31775 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:41.755007 31809 raft_consensus.cc:2366] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:41.745391 2405 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:41.764778 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:41.716984 2099 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.769913 31883 raft_consensus.cc:1238] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:41.771037 31803 raft_consensus.cc:2391] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:41.771073 31807 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:41.772219 2275 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:41.773880 1804 consensus_meta.cc:287] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.671s user 0.024s sys 0.015s
W20250831 07:59:41.776021 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:41.776487 2436 log.cc:927] Time spent T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20250831 07:59:41.777245 32470 raft_consensus.cc:1273] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:41.782596 31879 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
W20250831 07:59:41.776971 31698 leader_election.cc:336] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.907s (SENT)
I20250831 07:59:41.777478 2236 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.786763 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.746440 2350 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.789158 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 311) took 2105 ms. Trace:
I20250831 07:59:41.790166 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 310) took 2107 ms. Trace:
I20250831 07:59:41.786878 31729 raft_consensus.cc:2466] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
W20250831 07:59:41.794560 31879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1865) took 1563 ms (client timeout 1691 ms). Trace:
I20250831 07:59:41.795128 31883 rpcz_store.cc:276] 0831 07:59:39.683297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.746550 (+2063253us) service_pool.cc:225] Handling call
0831 07:59:41.769734 (+ 23184us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.788986 (+ 19252us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.789116 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.795838 31804 raft_consensus.cc:2391] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:41.795195 31875 rpcz_store.cc:276] 0831 07:59:39.683009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.705488 (+2022479us) service_pool.cc:225] Handling call
0831 07:59:41.750047 (+ 44559us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.789884 (+ 39837us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:41.789919 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:41.789971 (+ 52us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:41.790006 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.790061 (+ 55us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:41.790129 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:41.796516 31879 rpcz_store.cc:269] 0831 07:59:40.228604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.493253 (+1264649us) service_pool.cc:225] Handling call
0831 07:59:41.792033 (+298780us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.800956 32434 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:41.802285 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.765739 2348 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:41.808619 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.778648 2186 consensus_queue.cc:1046] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20250831 07:59:41.807643 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:41.805313 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 66495.5us
I20250831 07:59:41.812048 31874 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:41.815620 31801 raft_consensus.cc:2391] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 8: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:59:41.819479 2437 log.cc:927] Time spent T ccd51d2e599e4535ad0f2f6afcb1bfee P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20250831 07:59:41.820143 31806 raft_consensus.cc:2391] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:41.821571 2348 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:41.822619 31874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1875) took 1393 ms. Trace:
I20250831 07:59:41.822971 31874 rpcz_store.cc:276] 0831 07:59:40.429322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.570888 (+1141566us) service_pool.cc:225] Handling call
0831 07:59:41.822473 (+251585us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b54001edcd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:41.822570 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":479273856}
I20250831 07:59:41.824039 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.798691 2332 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:41.830719 32462 raft_consensus.cc:1273] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250831 07:59:41.793748 1804 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.831105 2348 raft_consensus.cc:3053] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:41.835655 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 315) took 1496 ms. Trace:
I20250831 07:59:41.816891 2286 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:41.837512 31874 raft_consensus.cc:2466] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 4.
I20250831 07:59:41.837230 31878 rpcz_store.cc:276] 0831 07:59:40.339556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.751754 (+1412198us) service_pool.cc:225] Handling call
0831 07:59:41.796847 (+ 45093us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.835285 (+ 38438us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:41.835326 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:41.835373 (+ 47us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:41.835415 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.835479 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:41.835589 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.836294 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.839743 31775 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:41.839883 31883 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:41.842104 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.841962 2201 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.843017 31882 raft_consensus.cc:1238] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:41.843561 31878 raft_consensus.cc:2466] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:41.836795 2348 raft_consensus.cc:738] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:41.845808 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 45145.7us
W20250831 07:59:41.851025 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.825145 1966 consensus_queue.cc:1046] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20250831 07:59:41.835232 2236 raft_consensus.cc:2802] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:41.857385 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.855988 32464 raft_consensus.cc:1238] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:41.856194 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:41.862769 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:41.859696 32434 leader_election.cc:336] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.880s (SENT)
I20250831 07:59:41.864161 32434 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:41.864169 31878 raft_consensus.cc:2366] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:41.812981 2275 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.870160 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:41.853940 2302 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:41.885149 2186 raft_consensus.cc:3053] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:41.885628 2186 raft_consensus.cc:738] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:41.886587 2186 consensus_queue.cc:260] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:41.871951 2096 raft_consensus.cc:2802] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250831 07:59:41.875948 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 99033.4us
I20250831 07:59:41.876745 2099 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.877272 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:41.877763 32462 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 419) took 3868 ms. Trace:
I20250831 07:59:41.859372 2236 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:41.880127 2148 consensus_meta.cc:287] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.731s user 0.004s sys 0.011s
W20250831 07:59:41.881071 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 68296.7us
I20250831 07:59:41.883664 31805 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 8.
I20250831 07:59:41.888706 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1476) took 2177 ms. Trace:
I20250831 07:59:41.889842 31807 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 8.
I20250831 07:59:41.881220 1947 raft_consensus.cc:2802] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:41.856159 2201 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:41.894953 31882 rpcz_store.cc:276] 0831 07:59:39.691428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.746584 (+2055156us) service_pool.cc:225] Handling call
0831 07:59:41.788696 (+ 42112us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.843367 (+ 54671us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.868937 (+ 25570us) spinlock_profiling.cc:243] Waited 54.1 ms on lock 0x7b5400058554. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:41.869137 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":124519296}
I20250831 07:59:41.902513 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.902338 2148 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:41.899835 31775 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:41.903316 1804 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.892278 2096 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:41.904460 31775 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:41.905663 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.899051 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 318) took 1558 ms. Trace:
I20250831 07:59:41.907639 31874 raft_consensus.cc:1273] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer a0d845d01c8c4894895f66e3beae298c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:41.907147 31879 rpcz_store.cc:276] 0831 07:59:40.340232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.801903 (+1461671us) service_pool.cc:225] Handling call
0831 07:59:41.829442 (+ 27539us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.891900 (+ 62458us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:41.891943 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:41.898709 (+ 6766us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:41.898775 (+ 66us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.898855 (+ 80us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:41.898986 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:41.902729 31878 raft_consensus.cc:1238] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8. Ops: []
I20250831 07:59:41.896844 2236 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:41.919317 31809 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:41.916630 31883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1483) took 1575 ms. Trace:
I20250831 07:59:41.924059 31883 rpcz_store.cc:276] 0831 07:59:40.340829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.854614 (+1513785us) service_pool.cc:225] Handling call
0831 07:59:41.865019 (+ 10405us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.909041 (+ 44022us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:41.909081 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:41.916439 (+ 7358us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:41.916473 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.916514 (+ 41us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:41.916588 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:41.927287 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 67626.9us
I20250831 07:59:41.896868 1966 raft_consensus.cc:2747] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:41.894544 32462 rpcz_store.cc:276] 0831 07:59:38.009904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:38.010050 (+ 146us) service_pool.cc:225] Handling call
0831 07:59:38.010416 (+ 366us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.843861 (+3833445us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:41.877510 (+ 33649us) spinlock_profiling.cc:243] Waited 3.75 s on lock 0x7b54005ad2d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:41.877690 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8614727552}
I20250831 07:59:41.907145 2096 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:41.929280 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.928424 2142 raft_consensus.cc:2802] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:41.933975 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.937956 2228 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 07:59:41.938470 2228 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:41.909374 1947 raft_consensus.cc:695] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:41.938985 2228 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:41.947155 31883 raft_consensus.cc:1238] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:41.930716 2186 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:41.953540 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 168669us
W20250831 07:59:41.936209 2279 consensus_meta.cc:287] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.794s user 0.009s sys 0.017s
W20250831 07:59:41.947870 2440 log.cc:927] Time spent T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20250831 07:59:41.931201 2148 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:41.963579 31773 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.965569 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:41.953794 2142 raft_consensus.cc:695] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [term 8 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:41.970676 31882 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:41.978415 31775 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:41.977916 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:41.980898 31883 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:41.960878 2332 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
W20250831 07:59:41.991619 31698 leader_election.cc:336] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.865s (SENT)
I20250831 07:59:41.987970 2391 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:41.990720 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:42.003001 32464 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Advancing to term 9
W20250831 07:59:42.004611 31698 leader_election.cc:336] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.910s (SENT)
W20250831 07:59:42.005311 31698 leader_election.cc:336] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.979s (SENT)
I20250831 07:59:42.005679 31698 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:42.006975 31698 leader_election.cc:336] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 2.186s (SENT)
I20250831 07:59:42.007386 31698 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:42.009382 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:42.010354 32462 raft_consensus.cc:2408] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20250831 07:59:42.010579 2241 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:42.015964 2332 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.018087 2332 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:42.018882 2391 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:42.020776 2391 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:42.021086 31699 connection.cc:232] RPC call timeout handler was delayed by -1.4762259549999044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
I20250831 07:59:41.970386 2279 raft_consensus.cc:2747] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250831 07:59:42.000218 2142 consensus_queue.cc:237] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.022279 31802 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 10.
I20250831 07:59:42.023463 31802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1755) took 1012 ms. Trace:
I20250831 07:59:42.023777 31802 rpcz_store.cc:276] 0831 07:59:41.011304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.022262 (+ 10958us) service_pool.cc:225] Handling call
0831 07:59:42.023286 (+1001024us) spinlock_profiling.cc:243] Waited 999 ms on lock 0x7b54000ce4d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:42.023404 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2296846080}
I20250831 07:59:42.008440 2341 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250831 07:59:42.025300 31699 connection.cc:232] RPC call timeout handler was delayed by -1.4141259549999177s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21800000000000003s for a response
I20250831 07:59:42.028425 2275 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.955090 2348 consensus_queue.cc:260] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.029608 2348 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:42.031909 31631 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43950 (request call id 123) took 10680 ms. Trace:
I20250831 07:59:42.035560 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:42.037005 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:42.037622 32463 raft_consensus.cc:2466] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:42.038305 2341 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:42.038852 2341 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.037420 2241 consensus_queue.cc:1046] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250831 07:59:42.040373 2405 raft_consensus.cc:3053] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:42.040756 2405 raft_consensus.cc:738] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:42.041581 2341 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:42.041972 2405 consensus_queue.cc:260] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:41.975524 2350 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:41.995361 31726 raft_consensus.cc:2466] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
W20250831 07:59:42.044919 31699 connection.cc:232] RPC call timeout handler was delayed by -1.3610259549999455s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2239s for a response
W20250831 07:59:42.049696 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 76839.4us
W20250831 07:59:42.050026 31699 connection.cc:232] RPC call timeout handler was delayed by -1.2314533959998926s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2919s for a response
W20250831 07:59:42.059298 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 60401.8us
I20250831 07:59:42.043329 2405 raft_consensus.cc:3058] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:42.059521 31881 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:42.065172 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 320) took 1057 ms. Trace:
I20250831 07:59:42.065538 31881 rpcz_store.cc:276] 0831 07:59:41.009946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.609816 (+599870us) service_pool.cc:225] Handling call
0831 07:59:42.065111 (+455295us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:42.066416 31880 raft_consensus.cc:1238] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:42.032354 31631 rpcz_store.cc:276] 0831 07:59:31.252707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:31.264017 (+ 11310us) service_pool.cc:225] Handling call
0831 07:59:41.927798 (+10663781us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 07:59:32.031398 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 07:59:32.220034 (+188636us) write_op.cc:432] Acquiring schema lock in shared mode
0831 07:59:32.227610 (+ 7576us) write_op.cc:435] Acquired schema lock
0831 07:59:32.241808 (+ 14198us) tablet.cc:660] Decoding operations
0831 07:59:32.242900 (+ 1092us) write_op.cc:620] Acquiring the partition lock for write op
0831 07:59:32.242978 (+ 78us) write_op.cc:641] Partition lock acquired for write op
0831 07:59:32.243006 (+ 28us) tablet.cc:683] Acquiring locks for 11 operations
0831 07:59:32.473514 (+230508us) tablet.cc:699] Row locks acquired
0831 07:59:32.506153 (+ 32639us) write_op.cc:260] PREPARE: finished
0831 07:59:32.537912 (+ 31759us) write_op.cc:270] Start()
0831 07:59:32.538099 (+ 187us) write_op.cc:276] Timestamp: P: 1756627172537823 usec, L: 0
0831 07:59:32.538129 (+ 30us) op_driver.cc:348] REPLICATION: starting
0831 07:59:32.696382 (+158253us) log.cc:844] Serialized 4330 byte log entry
0831 07:59:33.509054 (+812672us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:34.285885 (+776831us) op_driver.cc:464] REPLICATION: finished
0831 07:59:34.858620 (+572735us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce507e39c 00007fdcedcb9b89 00007fdce509aa4b 00007fdce5098cef 00007fdce509afb2 00007fdceba20e50
0831 07:59:35.197860 (+339240us) write_op.cc:301] APPLY: starting
0831 07:59:35.248332 (+ 50472us) tablet.cc:1365] starting BulkCheckPresence
0831 07:59:35.324639 (+ 76307us) tablet.cc:1368] finished BulkCheckPresence
0831 07:59:35.324674 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 07:59:35.451407 (+126733us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 07:59:35.451793 (+ 386us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0831 07:59:35.451890 (+ 97us) write_op.cc:312] APPLY: finished
0831 07:59:35.452270 (+ 380us) log.cc:844] Serialized 83 byte log entry
0831 07:59:35.470022 (+ 17752us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:35.491161 (+ 21139us) write_op.cc:454] Released schema lock
0831 07:59:35.495619 (+ 4458us) write_op.cc:341] FINISH: Updating metrics
0831 07:59:35.501060 (+ 5441us) write_op.cc:489] Releasing partition, row and schema locks
0831 07:59:35.501090 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":874487,"apply.run_cpu_time_us":8396,"apply.run_wall_time_us":326526,"num_ops":11,"prepare.queue_time_us":162562,"prepare.run_cpu_time_us":9316,"prepare.run_wall_time_us":824714,"raft.queue_time_us":526046,"raft.run_cpu_time_us":36719,"raft.run_wall_time_us":858269,"replication_time_us":1747604,"spinlock_wait_cycles":48695424,"thread_start_us":1353036,"threads_started":5,"wal-append.queue_time_us":120590,"wal-append.run_cpu_time_us":53286,"wal-append.run_wall_time_us":3397477}]]}
I20250831 07:59:42.069370 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.070124 31878 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
W20250831 07:59:42.065913 2392 consensus_meta.cc:287] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.772s user 0.011s sys 0.016s
W20250831 07:59:42.064939 2220 consensus_meta.cc:287] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.672s user 0.008s sys 0.021s
I20250831 07:59:42.066720 2148 raft_consensus.cc:2802] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
W20250831 07:59:42.077836 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:41.972862 1947 consensus_queue.cc:237] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.074900 2096 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.076874 2392 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.102736 2096 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:42.104007 2099 consensus_queue.cc:1046] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250831 07:59:42.088371 31775 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.092691 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.091715 31877 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 10.
I20250831 07:59:42.092001 31880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1854) took 2214 ms. Trace:
I20250831 07:59:42.106364 31880 rpcz_store.cc:276] 0831 07:59:39.872444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.737401 (+1864957us) service_pool.cc:225] Handling call
0831 07:59:42.066236 (+328835us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:42.066824 (+ 588us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:42.087010 (+ 20186us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b54002bd250. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:42.087143 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":545701248}
I20250831 07:59:42.090003 2148 raft_consensus.cc:695] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:42.066015 32464 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 9.
I20250831 07:59:42.110988 1843 consensus_queue.cc:1035] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20250831 07:59:42.111841 2392 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:42.111745 2236 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.112752 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.113481 31880 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 7: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:42.113857 2236 leader_election.cc:290] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:42.114277 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:42.114837 31880 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:42.116103 31881 raft_consensus.cc:2391] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 8: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:42.115679 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.086755 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:42.119398 31880 raft_consensus.cc:2391] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 7: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:42.121114 31876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1475) took 2430 ms. Trace:
I20250831 07:59:42.119777 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:42.126843 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:42.125646 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 50509us
I20250831 07:59:42.121735 31876 rpcz_store.cc:276] 0831 07:59:39.691049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.736455 (+2045406us) service_pool.cc:225] Handling call
0831 07:59:41.766197 (+ 29742us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.802349 (+ 36152us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 07:59:41.969157 (+166808us) log.cc:844] Serialized 14 byte log entry
0831 07:59:42.120687 (+151530us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:42.120760 (+ 73us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:42.120817 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:42.120904 (+ 87us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:42.121048 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":192644,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":45,"thread_start_us":243128,"threads_started":2}
I20250831 07:59:42.132861 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 317) took 1793 ms. Trace:
I20250831 07:59:42.133963 31875 rpcz_store.cc:276] 0831 07:59:40.339940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:41.800413 (+1460473us) service_pool.cc:225] Handling call
0831 07:59:41.815401 (+ 14988us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:41.818408 (+ 3007us) raft_consensus.cc:1532] Early marking committed up to index 9
0831 07:59:41.958906 (+140498us) log.cc:844] Serialized 14 byte log entry
0831 07:59:42.132511 (+173605us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:42.132569 (+ 58us) raft_consensus.cc:1643] Marking committed up to 9
0831 07:59:42.132619 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:42.132682 (+ 63us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:42.132802 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15447,"prepare.queue_time_us":180494,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"thread_start_us":258420,"threads_started":2}
I20250831 07:59:42.136230 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:42.137351 31875 raft_consensus.cc:2408] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250831 07:59:42.138784 31875 raft_consensus.cc:1238] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 3. Current term is 5. Ops: []
I20250831 07:59:42.141781 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:42.149909 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 164307us
I20250831 07:59:42.091840 2350 consensus_queue.cc:1035] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20250831 07:59:42.091696 31628 catalog_manager.cc:5582] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 4 to 7, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 7 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
W20250831 07:59:42.159417 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 50690.2us
I20250831 07:59:42.159868 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.156523 31881 raft_consensus.cc:3058] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:42.162282 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 119004us
I20250831 07:59:42.162593 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.169819 31876 raft_consensus.cc:2391] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 7: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:42.107730 2341 consensus_queue.cc:1046] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20250831 07:59:42.156702 2392 raft_consensus.cc:3053] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Stepping down as leader of term 6
W20250831 07:59:42.178408 31877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1456) took 3022 ms (client timeout 1891 ms). Trace:
W20250831 07:59:42.178810 31877 rpcz_store.cc:269] 0831 07:59:39.155891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:39.156572 (+ 681us) service_pool.cc:225] Handling call
0831 07:59:42.168998 (+3012426us) spinlock_profiling.cc:243] Waited 2.91 s on lock 0x7b540020c6d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:42.178353 (+ 9355us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6693853440}
W20250831 07:59:42.184167 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:42.161885 1843 raft_consensus.cc:2802] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:42.200088 2241 raft_consensus.cc:3053] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:42.200645 2241 raft_consensus.cc:738] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:42.201851 2148 consensus_queue.cc:237] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.205204 2220 consensus_queue.cc:1035] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20250831 07:59:42.204941 31628 catalog_manager.cc:5582] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 3 to 6, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 6 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:59:42.206632 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:42.206734 31628 catalog_manager.cc:5582] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:59:42.207407 32462 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8.
I20250831 07:59:42.186342 31849 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:42.201468 2241 consensus_queue.cc:260] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.190812 2142 consensus_queue.cc:1035] T 2acd7b3ac5f94f53ac1ef93264415686 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20250831 07:59:42.214731 2220 raft_consensus.cc:2747] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:42.214509 2241 raft_consensus.cc:3058] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:42.219100 31628 catalog_manager.cc:5582] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:59:42.219718 32469 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:42.220609 31881 raft_consensus.cc:2466] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
W20250831 07:59:42.219290 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 45916.2us
W20250831 07:59:42.240438 31698 leader_election.cc:336] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.522s (SENT)
I20250831 07:59:42.243949 31809 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
I20250831 07:59:42.177412 2392 raft_consensus.cc:738] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
W20250831 07:59:42.256023 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:42.259111 32435 leader_election.cc:304] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:42.260376 1966 raft_consensus.cc:2802] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:42.261060 1966 raft_consensus.cc:695] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [term 7 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:42.262238 1966 consensus_queue.cc:237] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.234088 1843 raft_consensus.cc:695] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:42.276221 31628 catalog_manager.cc:5582] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 2 to 4, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 4 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:59:42.281780 2228 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:42.287070 31882 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:42.290467 31849 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:42.298738 31699 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.299659 2392 consensus_queue.cc:260] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.299388 1843 consensus_queue.cc:237] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.301047 2392 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:42.308045 31880 raft_consensus.cc:2466] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:42.308841 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:42.308403 32434 leader_election.cc:336] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 2.100s (SENT)
I20250831 07:59:42.312418 32434 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:42.314226 2455 raft_consensus.cc:2762] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250831 07:59:42.314592 31699 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.315629 2455 raft_consensus.cc:2762] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250831 07:59:42.317402 31699 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:42.318818 31699 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.319339 2455 raft_consensus.cc:2747] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:42.317377 31847 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:42.321991 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:42.304881 31628 catalog_manager.cc:5582] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
W20250831 07:59:42.325826 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 134317us
I20250831 07:59:42.326052 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:42.327159 31729 raft_consensus.cc:2391] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:42.323904 2455 raft_consensus.cc:2802] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader election won for term 9
I20250831 07:59:42.320607 2391 raft_consensus.cc:2747] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:42.333240 31628 catalog_manager.cc:5582] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:59:42.336177 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:42.336381 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:42.336658 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:42.337524 2186 raft_consensus.cc:2747] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 07:59:42.340283 31726 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Advancing to term 8
W20250831 07:59:42.339562 2451 log.cc:927] Time spent T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20250831 07:59:42.341408 2228 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:59:42.343330 32469 raft_consensus.cc:1238] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:42.296620 2405 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:42.344206 2405 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.344830 2275 consensus_queue.cc:1046] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250831 07:59:42.345849 2275 raft_consensus.cc:3053] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:42.346060 2405 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:42.346135 2275 raft_consensus.cc:738] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:42.347241 31726 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 8.
I20250831 07:59:42.347259 2275 consensus_queue.cc:260] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.349179 2458 raft_consensus.cc:2747] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:42.350347 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:42.348876 31628 catalog_manager.cc:5582] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 3 to 4, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 4 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:59:42.350922 31804 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
W20250831 07:59:42.356786 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 77631.6us
I20250831 07:59:42.348834 2275 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:42.363073 2455 raft_consensus.cc:695] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 9 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:42.366592 31849 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.367630 31849 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:42.367323 2455 consensus_queue.cc:237] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:42.368906 31849 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.369927 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:42.370759 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 53593.2us
I20250831 07:59:42.371539 31847 leader_election.cc:304] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:42.372723 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:42.373327 31729 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:42.374528 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:42.376001 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:42.376664 31732 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 7
I20250831 07:59:42.385998 31733 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:42.388437 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:42.389505 31882 raft_consensus.cc:2391] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:42.392279 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:42.373994 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:42.384891 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:42.391654 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:42.401827 31698 leader_election.cc:336] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.600s (SENT)
I20250831 07:59:42.402456 31698 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:42.404131 31775 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.406863 31775 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.405599 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:42.399374 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:42.380348 2391 raft_consensus.cc:2747] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250831 07:59:42.394330 2286 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:42.413715 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:42.380501 2458 raft_consensus.cc:2802] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:42.414482 31731 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:42.411121 31628 catalog_manager.cc:5582] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 3 to 6, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 6 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:59:42.413604 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:42.421386 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 49934.7us
I20250831 07:59:42.421774 31876 raft_consensus.cc:1273] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 07:59:42.420703 31728 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 8.
I20250831 07:59:42.423467 31805 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8.
I20250831 07:59:42.433900 31775 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.411509 2148 raft_consensus.cc:2802] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:42.438829 2148 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:42.439205 2148 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:42.424991 2341 raft_consensus.cc:2747] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:42.436007 2452 raft_consensus.cc:2747] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:42.425169 2099 raft_consensus.cc:2802] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:42.444787 2099 raft_consensus.cc:695] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 8 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:42.445726 2099 consensus_queue.cc:237] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.447988 31801 raft_consensus.cc:2366] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:42.422345 2286 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:42.448840 2286 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:42.439422 1947 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:42.450412 1947 raft_consensus.cc:695] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:42.435542 1804 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:42.451939 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:42.452303 1804 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.453488 31628 catalog_manager.cc:5582] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 3 to 4, leader changed from 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2) to a0d845d01c8c4894895f66e3beae298c (127.29.217.1). New cstate: current_term: 4 leader_uuid: "a0d845d01c8c4894895f66e3beae298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
I20250831 07:59:42.454972 1804 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:42.456372 31847 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:42.457876 31847 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:42.458940 2391 raft_consensus.cc:2747] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 07:59:42.460168 31847 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:42.459565 2391 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:42.461117 2391 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:42.461365 31847 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:42.462823 31847 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:42.463001 2448 raft_consensus.cc:2747] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:42.463989 2391 raft_consensus.cc:2802] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:42.464372 2391 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:42.464687 2391 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:42.451320 1947 consensus_queue.cc:237] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.455240 31628 catalog_manager.cc:5582] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 3 to 5, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 5 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:59:42.473582 31847 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:42.474609 31731 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:42.474906 31729 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:42.476070 31775 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.477276 31732 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:42.480368 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.481177 31882 raft_consensus.cc:3058] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:42.484678 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.486500 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.487385 31880 raft_consensus.cc:3058] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:42.487973 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:42.488639 31727 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:42.492341 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:42.503031 32469 raft_consensus.cc:1238] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 8. Current term is 9. Ops: []
I20250831 07:59:42.506289 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:42.506925 31881 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8.
I20250831 07:59:42.507491 31880 raft_consensus.cc:2466] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:42.425586 2458 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:42.508729 31628 catalog_manager.cc:5582] T be7a7e4ea6e1431ab37fb092f87cebd3 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 4 to 5, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 5 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:59:42.511236 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:42.500556 31874 raft_consensus.cc:2374] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7. Re-sending same reply.
I20250831 07:59:42.514015 31775 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.520954 2391 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.523133 2228 raft_consensus.cc:2802] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:42.523304 2391 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:42.523726 2228 raft_consensus.cc:695] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:42.524098 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.524888 31876 raft_consensus.cc:2391] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 8: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:42.524622 31628 catalog_manager.cc:5582] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 5, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 5 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:59:42.452986 2457 consensus_queue.cc:1035] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.105s
I20250831 07:59:42.526254 31628 catalog_manager.cc:5582] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 5, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 5 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
W20250831 07:59:42.528018 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 149964us
I20250831 07:59:42.528807 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.527746 31628 catalog_manager.cc:5582] T b606cb7b5ccc4e2588ca98715c021620 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 6, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 6 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:59:42.532127 31775 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:42.540056 31698 leader_election.cc:336] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.662s (SENT)
I20250831 07:59:42.500559 2452 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:42.541791 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:42.530736 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:42.546180 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.547385 31883 raft_consensus.cc:2408] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250831 07:59:42.543741 31628 catalog_manager.cc:5582] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 3 to 6, leader changed from 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3) to 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2). New cstate: current_term: 6 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20250831 07:59:42.552002 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.512447 2458 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:42.529589 2275 raft_consensus.cc:2802] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:42.548148 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:42.524904 2228 consensus_queue.cc:237] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.557955 31775 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.559876 2142 raft_consensus.cc:2802] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:42.561923 31628 catalog_manager.cc:5582] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "7512ba2ed3a94982a22a75f51a647e65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20250831 07:59:42.543665 2452 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:42.579918 31698 leader_election.cc:336] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.681s (SENT)
I20250831 07:59:42.580837 31698 leader_election.cc:304] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:42.580051 31876 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8.
I20250831 07:59:42.564256 31882 raft_consensus.cc:2466] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:42.594990 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.595692 31877 raft_consensus.cc:3058] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:42.595911 2241 raft_consensus.cc:2747] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:42.598505 31628 catalog_manager.cc:5582] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 reported cstate change: term changed from 2 to 5, leader changed from a0d845d01c8c4894895f66e3beae298c (127.29.217.1) to 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3). New cstate: current_term: 5 leader_uuid: "8247a38b550649aa9cd6e7668dcaec10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } }
W20250831 07:59:42.599592 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 227355us
W20250831 07:59:42.607465 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 44763.6us
I20250831 07:59:42.609094 31728 raft_consensus.cc:1273] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 07:59:42.581887 2452 raft_consensus.cc:3058] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:42.562875 2391 consensus_queue.cc:1046] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
W20250831 07:59:42.613308 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:42.615118 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:42.616151 31882 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:42.564157 2286 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.620693 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:42.621551 31876 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:42.615932 2099 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:42.635681 2302 consensus_queue.cc:1035] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:59:42.644273 2099 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:42.640966 2286 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:42.642598 31877 raft_consensus.cc:2466] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
W20250831 07:59:42.646620 2469 log.cc:927] Time spent T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20250831 07:59:42.652066 2099 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:42.660599 31808 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250831 07:59:42.638836 31728 raft_consensus.cc:1273] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 07:59:42.662106 31699 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:42.629530 2275 raft_consensus.cc:695] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:42.668454 2275 consensus_queue.cc:237] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.629724 2142 raft_consensus.cc:695] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:42.675267 2096 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250831 07:59:42.638347 2148 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:42.684156 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 60896.3us
I20250831 07:59:42.685076 2142 consensus_queue.cc:237] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.685987 2148 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:42.688123 2448 raft_consensus.cc:3053] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 07:59:42.690032 31882 raft_consensus.cc:2408] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250831 07:59:42.685091 31699 leader_election.cc:304] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:42.688571 2448 raft_consensus.cc:738] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:42.716777 2448 consensus_queue.cc:260] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.718840 2448 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:42.706772 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:42.730180 31727 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:42.688859 2457 raft_consensus.cc:2747] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250831 07:59:42.739691 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:42.740017 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 107791us
W20250831 07:59:42.743963 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:42.744282 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 59305.7us
W20250831 07:59:42.746662 31698 leader_election.cc:336] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.790s (SENT)
I20250831 07:59:42.696606 2452 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.728662 2471 raft_consensus.cc:2747] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:42.753222 31698 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:42.704128 1843 consensus_queue.cc:1035] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20250831 07:59:42.719592 2099 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.762342 31732 raft_consensus.cc:1273] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 07:59:42.773737 2452 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:42.774942 31699 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:42.776705 31699 leader_election.cc:304] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:42.772003 32436 leader_election.cc:336] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.512s (SENT)
I20250831 07:59:42.777508 32436 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:42.781153 2471 raft_consensus.cc:2747] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 07:59:42.776999 2458 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:42.782699 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:42.762652 2457 raft_consensus.cc:2747] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 07:59:42.785049 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 46999.6us
W20250831 07:59:42.795800 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 49719.8us
I20250831 07:59:42.792474 2471 raft_consensus.cc:2747] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250831 07:59:42.804379 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:42.808128 2458 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:42.808769 31848 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:42.814512 32470 raft_consensus.cc:2466] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:42.795033 2348 raft_consensus.cc:2747] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250831 07:59:42.822120 31847 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:42.826668 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:42.834831 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:42.789374 2228 consensus_queue.cc:1035] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
W20250831 07:59:42.860927 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 46350.3us
W20250831 07:59:42.862550 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 47619.2us
W20250831 07:59:42.864176 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 87553.3us
I20250831 07:59:42.866742 31699 leader_election.cc:400] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:42.867131 31699 leader_election.cc:403] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250831 07:59:42.871234 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:42.872133 31729 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:42.827654 2458 raft_consensus.cc:2802] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:42.830341 2452 raft_consensus.cc:2802] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:42.881772 31734 raft_consensus.cc:1273] T 14ab15c60aaf42cf911851425eaab94d P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 07:59:42.882041 31699 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.884116 31848 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:42.879406 2458 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:42.889727 31729 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 8.
I20250831 07:59:42.891151 31775 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.891527 31699 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:42.878795 2241 consensus_queue.cc:1035] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20250831 07:59:42.897214 2286 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:42.878603 2341 consensus_queue.cc:1035] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20250831 07:59:42.897822 2286 raft_consensus.cc:695] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:42.898895 2286 consensus_queue.cc:237] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.887569 2474 consensus_queue.cc:1035] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20250831 07:59:42.891547 2099 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:42.884945 2452 raft_consensus.cc:695] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:42.888684 2454 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250831 07:59:42.927838 31699 leader_election.cc:304] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:42.886605 2455 raft_consensus.cc:2747] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
W20250831 07:59:42.927100 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:42.938784 2148 raft_consensus.cc:2802] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:42.958359 31727 raft_consensus.cc:2466] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:42.958763 2148 raft_consensus.cc:695] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [term 8 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:42.959625 2148 consensus_queue.cc:237] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:42.931964 2454 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:42.964259 2454 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 10 FOLLOWER]: Advancing to term 11
I20250831 07:59:42.929078 1843 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:42.931725 31848 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:42.952580 31699 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:42.969209 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 47332us
I20250831 07:59:42.953207 2471 raft_consensus.cc:2747] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:42.973110 2220 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 07:59:42.973565 2220 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:42.973930 2220 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:42.975912 2457 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 07:59:42.976334 2457 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:42.957293 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:42.890901 2458 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:42.963776 2241 raft_consensus.cc:2762] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250831 07:59:42.949018 2452 consensus_queue.cc:237] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.968905 1843 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:42.976645 2457 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:42.970571 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:42.968318 31849 leader_election.cc:336] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.942s (SENT)
I20250831 07:59:42.971338 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:42.972290 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 49636.6us
I20250831 07:59:42.969568 31699 leader_election.cc:400] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:43.020293 31882 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:43.020615 31699 leader_election.cc:403] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250831 07:59:42.994482 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:42.995620 31775 leader_election.cc:304] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:42.954223 2481 log.cc:927] Time spent T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.072s user 0.002s sys 0.000s
W20250831 07:59:43.035293 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 65869.3us
W20250831 07:59:43.053530 31772 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:43.020386 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 158834us
W20250831 07:59:43.018105 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:43.035003 2275 raft_consensus.cc:2802] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:43.059015 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.059759 31733 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
W20250831 07:59:43.065310 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 95193.1us
I20250831 07:59:43.064431 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.060709 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.066924 31734 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:43.067508 31733 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
W20250831 07:59:43.068078 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 145511us
I20250831 07:59:43.079279 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:43.085418 2241 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:43.086045 2241 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.092077 2096 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:43.092661 2096 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:43.095283 2096 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:43.061249 31730 raft_consensus.cc:1215] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Deduplicated request from leader. Original: 4.6->[6.7-6.7] Dedup: 6.7->[]
I20250831 07:59:43.130787 2471 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:43.144558 31730 raft_consensus.cc:1238] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250831 07:59:43.165534 2241 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:43.189499 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:43.060148 1843 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:43.068874 2275 raft_consensus.cc:695] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:43.250864 2448 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:43.251528 2448 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:43.252178 2457 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.253580 2448 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:43.258071 2471 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261) with higher term. Message: Invalid argument: T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:43.259388 2458 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.261677 2458 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:43.267108 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 298994us
W20250831 07:59:43.269052 32436 leader_election.cc:336] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.828s (SENT)
W20250831 07:59:43.270200 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:43.267526 31698 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:43.271397 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:43.270850 32434 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.520s (SENT) [suppressed 66 similar messages]
I20250831 07:59:43.275259 31734 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 8. Current term is 9. Ops: []
I20250831 07:59:43.268726 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:43.276088 31849 leader_election.cc:336] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.994s (SENT)
I20250831 07:59:43.276562 31849 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:43.280434 31734 raft_consensus.cc:1273] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 07:59:43.279325 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.282124 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.282471 31726 raft_consensus.cc:2374] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 9. Re-sending same reply.
I20250831 07:59:43.283854 31734 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 8. Current term is 9. Ops: []
I20250831 07:59:43.284212 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.289541 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.301256 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:43.302997 2454 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.303992 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:43.305819 32436 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:43.305373 2275 consensus_queue.cc:237] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.305977 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.306665 2220 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.308117 1843 consensus_queue.cc:1035] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:59:43.308780 2220 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:43.309324 32434 leader_election.cc:304] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:43.279877 2452 raft_consensus.cc:2747] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 07:59:43.314971 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 47569.8us
I20250831 07:59:43.284839 2241 raft_consensus.cc:2802] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 07:59:43.328014 2241 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:43.328641 2241 raft_consensus.cc:3058] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:43.310719 2332 raft_consensus.cc:2802] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:43.329496 2332 raft_consensus.cc:695] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [term 5 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:43.317569 1843 consensus_queue.cc:1046] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:59:43.330554 2457 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:43.331277 1843 raft_consensus.cc:3053] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 07:59:43.331843 1843 raft_consensus.cc:738] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:43.333247 1843 consensus_queue.cc:260] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.335546 1843 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:43.331440 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.336498 31732 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:43.337520 31725 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:43.344076 31734 raft_consensus.cc:2466] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
I20250831 07:59:43.345762 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:43.345906 31773 leader_election.cc:336] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585): Timed out: RequestConsensusVote RPC to 127.29.217.3:42585 timed out after 1.634s (SENT)
I20250831 07:59:43.359762 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.360916 31726 raft_consensus.cc:1273] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250831 07:59:43.360353 31808 raft_consensus.cc:2391] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 10: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:43.363153 31698 leader_election.cc:400] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Vote denied by peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585) with higher term. Message: Invalid argument: T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:43.364084 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.376454 31804 raft_consensus.cc:1238] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [term 8 LEADER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 7. Current term is 8. Ops: []
I20250831 07:59:43.363997 31773 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:43.379544 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.385459 31734 raft_consensus.cc:1273] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20250831 07:59:43.385720 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 48400us
I20250831 07:59:43.385785 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.389767 2099 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:43.402108 31730 raft_consensus.cc:1273] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 07:59:43.330618 2332 consensus_queue.cc:237] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:43.407745 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:43.408048 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:43.408797 31809 raft_consensus.cc:2391] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 7: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:43.404582 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:43.407155 1966 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:43.411901 2350 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:43.412179 2335 consensus_queue.cc:1046] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:59:43.412647 2350 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.413758 2335 raft_consensus.cc:3053] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:43.414111 2335 raft_consensus.cc:738] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:43.415378 2350 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:43.415139 31805 raft_consensus.cc:1273] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:43.416131 2335 consensus_queue.cc:260] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.412077 1966 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:43.424896 2454 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 11 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:43.425683 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.437367 31773 leader_election.cc:304] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:43.438408 2452 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:43.438926 2452 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:43.439697 31875 raft_consensus.cc:1273] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 07:59:43.439913 31698 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:43.440907 2452 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:43.440909 31698 leader_election.cc:400] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Vote denied by peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585) with higher term. Message: Invalid argument: T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8.
I20250831 07:59:43.442752 31809 raft_consensus.cc:1238] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 6. Current term is 7. Ops: []
W20250831 07:59:43.435741 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:43.447008 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.446877 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:43.448884 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:43.449903 31882 raft_consensus.cc:3058] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:43.454762 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:43.455798 31880 raft_consensus.cc:2391] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 8: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:43.459365 31731 raft_consensus.cc:3053] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:43.459525 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f70249b7dd14c0e804b05303fa06f09" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:43.459810 31731 raft_consensus.cc:738] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:43.460960 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.461593 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:43.464067 31849 leader_election.cc:336] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.721s (SENT)
I20250831 07:59:43.464148 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:43.464386 31883 raft_consensus.cc:1273] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 07:59:43.464581 31849 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:43.464493 31773 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:43.473299 32464 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:43.474488 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:43.475363 31809 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:43.477649 31804 raft_consensus.cc:1238] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 LEADER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:43.480057 31877 raft_consensus.cc:1273] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 07:59:43.480242 31801 raft_consensus.cc:1273] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250831 07:59:43.480512 31732 raft_consensus.cc:1273] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250831 07:59:43.484609 32434 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:43.484741 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:43.485307 32470 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 9.
I20250831 07:59:43.485445 2348 raft_consensus.cc:2747] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:43.479424 2228 raft_consensus.cc:2747] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:43.487038 31874 raft_consensus.cc:1238] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 8. Current term is 9. Ops: []
I20250831 07:59:43.488090 31802 raft_consensus.cc:2391] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:43.473196 31731 consensus_queue.cc:260] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.489279 31879 raft_consensus.cc:1273] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 07:59:43.489773 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.490456 31876 raft_consensus.cc:3053] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 07:59:43.490767 31876 raft_consensus.cc:738] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:43.493672 31848 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:43.460151 2454 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:43.494477 31847 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:43.499116 31878 raft_consensus.cc:1273] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 07:59:43.501086 31882 raft_consensus.cc:2466] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 8.
I20250831 07:59:43.489144 2220 raft_consensus.cc:2747] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:43.506242 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14ab15c60aaf42cf911851425eaab94d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.486582 2335 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:43.510740 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.511567 31732 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:43.512478 31731 raft_consensus.cc:3058] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:43.506827 31876 consensus_queue.cc:260] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.514727 31876 raft_consensus.cc:3058] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:43.499379 2302 raft_consensus.cc:2747] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:43.523353 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.519078 31698 leader_election.cc:400] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Vote denied by peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585) with higher term. Message: Invalid argument: T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
I20250831 07:59:43.522158 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.535984 31732 raft_consensus.cc:1238] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250831 07:59:43.539150 31725 raft_consensus.cc:1238] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:43.503857 2286 raft_consensus.cc:2747] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250831 07:59:43.539173 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 74758.6us
I20250831 07:59:43.502353 2454 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:43.557917 2496 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a9e73e8611540ee91fdee861c75c975: Append to log took a long time: real 0.258s user 0.005s sys 0.001s
I20250831 07:59:43.521624 2148 consensus_queue.cc:1035] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
W20250831 07:59:43.541622 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 93435.4us
I20250831 07:59:43.564926 31698 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:43.543653 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.515913 2228 consensus_queue.cc:1035] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20250831 07:59:43.554932 1966 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:43.516245 2448 consensus_queue.cc:1162] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7512ba2ed3a94982a22a75f51a647e65" responder_term: 8 status { last_received { term: 8 index: 5 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250831 07:59:43.556906 2241 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:43.593247 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 182791us
W20250831 07:59:43.594897 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 48306.8us
I20250831 07:59:43.595315 31731 raft_consensus.cc:1273] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:43.598562 31809 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:43.612845 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.596628 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:43.613926 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.545686 2220 raft_consensus.cc:2802] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:43.558763 2454 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:43.617743 31802 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:43.616133 31698 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:43.620640 2471 raft_consensus.cc:2762] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250831 07:59:43.617089 31733 raft_consensus.cc:1238] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 8. Current term is 9. Ops: []
I20250831 07:59:43.625543 31698 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:43.631152 31698 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:43.632055 2471 raft_consensus.cc:2802] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:43.592680 2457 raft_consensus.cc:2747] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:43.619035 32435 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
I20250831 07:59:43.641413 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.546277 2452 consensus_queue.cc:1046] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
W20250831 07:59:43.637568 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 143152us
W20250831 07:59:43.639142 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 46477us
I20250831 07:59:43.518321 32463 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0"
dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
mode: GRACEFUL
new_leader_uuid: "7512ba2ed3a94982a22a75f51a647e65"
from {username='slave'} at 127.0.0.1:46838
I20250831 07:59:43.645867 32463 raft_consensus.cc:604] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Received request to transfer leadership to TS 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:43.646311 32463 raft_consensus.cc:612] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 07:59:43.617045 2220 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:43.647365 2220 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:43.648034 31773 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:43.650354 2286 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:43.651023 2286 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:43.651919 2241 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:43.652534 2241 raft_consensus.cc:2747] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:43.653359 30564 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250831 07:59:43.655052 30564 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 07:59:43.642724 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.656698 31731 raft_consensus.cc:2374] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7. Re-sending same reply.
I20250831 07:59:43.612033 1966 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:43.657542 1966 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:43.657570 2504 raft_consensus.cc:2802] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:43.658883 2504 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:43.659204 2504 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:43.659469 1966 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:43.663009 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:59:43.640174 2471 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:43.665256 2471 raft_consensus.cc:3058] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:43.668399 2148 raft_consensus.cc:2802] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:43.668933 2148 raft_consensus.cc:695] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:43.669783 2148 consensus_queue.cc:237] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.653221 2286 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:43.672079 31808 raft_consensus.cc:2408] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20250831 07:59:43.672006 2452 consensus_queue.cc:1230] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer 7512ba2ed3a94982a22a75f51a647e65 log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20250831 07:59:43.671046 31849 leader_election.cc:336] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.198s (SENT)
I20250831 07:59:43.675299 31849 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:43.674407 31773 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:43.659641 31876 raft_consensus.cc:1273] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20250831 07:59:43.680066 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 170106us
I20250831 07:59:43.686269 31773 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:43.687656 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 191198us
I20250831 07:59:43.692306 2275 consensus_queue.cc:1035] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 07:59:43.692587 2220 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:43.693284 31773 leader_election.cc:304] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:43.693699 2275 raft_consensus.cc:2747] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 07:59:43.694332 2275 raft_consensus.cc:2802] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:43.694783 2275 raft_consensus.cc:695] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:43.694903 2350 raft_consensus.cc:3058] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:43.695958 2286 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:43.696276 2228 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:43.694765 2220 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:43.696501 2286 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.696815 2228 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:43.697360 2220 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:43.697924 2220 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.698473 2286 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:43.695801 2275 consensus_queue.cc:237] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.699082 2228 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:43.703655 31773 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:43.715097 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.720789 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:43.721139 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.721534 32463 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:43.722558 31847 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:43.723754 31733 raft_consensus.cc:2391] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 9: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
W20250831 07:59:43.726876 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 55961.2us
I20250831 07:59:43.727035 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:43.727532 31849 leader_election.cc:336] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.203s (SENT)
I20250831 07:59:43.727670 32463 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
W20250831 07:59:43.729256 31849 leader_election.cc:336] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 1.863s (SENT)
I20250831 07:59:43.729830 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:43.721030 2509 raft_consensus.cc:2747] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250831 07:59:43.728855 2302 consensus_queue.cc:1035] T 855219eabb9f4417ad63ec7efa7982da P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 07:59:43.714025 2448 raft_consensus.cc:2747] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:43.723286 2475 consensus_queue.cc:1035] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250831 07:59:43.735278 2458 raft_consensus.cc:3053] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:43.735685 2458 raft_consensus.cc:738] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:43.736617 2508 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:43.736496 2458 consensus_queue.cc:260] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.737094 2508 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.737481 2458 raft_consensus.cc:3058] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:43.739219 2508 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:43.733729 2099 raft_consensus.cc:2747] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:43.748385 31733 raft_consensus.cc:3053] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Stepping down as leader of term 7
I20250831 07:59:43.748936 31733 raft_consensus.cc:738] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:59:43.743769 31849 leader_election.cc:336] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027): Timed out: RequestConsensusVote RPC to 127.29.217.1:45027 timed out after 2.338s (SENT)
I20250831 07:59:43.727469 2220 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:43.770171 32463 raft_consensus.cc:1273] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250831 07:59:43.739727 2508 consensus_queue.cc:1046] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20250831 07:59:43.759625 2473 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:43.761878 2506 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:43.771713 31733 consensus_queue.cc:260] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.723665 2286 consensus_queue.cc:1035] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
W20250831 07:59:43.780176 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 73083.9us
I20250831 07:59:43.787968 31848 leader_election.cc:304] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:43.780977 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.782188 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.784168 31733 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Advancing to term 8
W20250831 07:59:43.785148 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:43.788806 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "855219eabb9f4417ad63ec7efa7982da" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.788741 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.770485 2448 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:43.797143 2448 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:43.798496 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.799283 2448 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:43.799611 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.800467 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:43.789837 31730 raft_consensus.cc:2374] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7. Re-sending same reply.
I20250831 07:59:43.802899 2454 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.805683 2099 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:43.806183 2099 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:43.791992 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:43.807992 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 176947us
I20250831 07:59:43.785876 2473 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:43.808779 31698 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:43.809975 31698 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:43.810371 2473 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:43.811024 31698 leader_election.cc:304] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:43.812041 2457 raft_consensus.cc:2802] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:43.812443 2457 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:43.812805 2457 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:43.813439 31729 raft_consensus.cc:1238] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Rejecting Update request from peer 8247a38b550649aa9cd6e7668dcaec10 for earlier term 7. Current term is 8. Ops: []
I20250831 07:59:43.772888 2220 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:43.816011 2241 raft_consensus.cc:2802] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:43.816447 2241 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:43.816207 2220 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:43.816819 2241 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:43.820962 2286 raft_consensus.cc:2802] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 07:59:43.821527 2286 raft_consensus.cc:695] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:43.822374 2286 consensus_queue.cc:237] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.789919 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.825444 31732 raft_consensus.cc:2374] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5. Re-sending same reply.
I20250831 07:59:43.771260 31849 leader_election.cc:304] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:43.796483 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:43.828538 31728 raft_consensus.cc:2374] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7. Re-sending same reply.
W20250831 07:59:43.835129 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 47982.5us
I20250831 07:59:43.783571 2506 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:43.846288 31733 raft_consensus.cc:1273] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20250831 07:59:43.847375 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 105468us
I20250831 07:59:43.849112 31773 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:43.786254 2508 consensus_queue.cc:1230] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Peer 7512ba2ed3a94982a22a75f51a647e65 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250831 07:59:43.857201 31773 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae; no voters:
W20250831 07:59:43.859437 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 131600us
I20250831 07:59:43.862594 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:43.863566 32463 raft_consensus.cc:2408] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20250831 07:59:43.834568 2513 raft_consensus.cc:2762] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250831 07:59:43.868379 2099 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:43.854485 2506 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:43.828835 2448 consensus_queue.cc:1035] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20250831 07:59:43.877591 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:43.877960 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:43.878278 32463 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:43.864629 2302 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:43.879354 2220 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:43.879506 2302 raft_consensus.cc:491] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:43.879951 2302 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:43.860628 2392 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20250831 07:59:43.884752 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:43.889247 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:43.857339 2471 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.861275 2348 consensus_queue.cc:1046] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.029s
I20250831 07:59:43.886843 2452 raft_consensus.cc:3053] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:43.887099 2392 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:43.886657 2508 raft_consensus.cc:2747] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:43.883937 2516 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:43.920426 2471 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:43.921350 2392 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:43.933015 31698 leader_election.cc:400] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Vote denied by peer 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585) with higher term. Message: Invalid argument: T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7.
W20250831 07:59:43.962096 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 174267us
I20250831 07:59:43.964179 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:43.965767 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:43.966612 31882 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:43.968287 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:43.968982 31882 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:43.912364 2452 raft_consensus.cc:738] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:43.980091 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:43.980939 31802 raft_consensus.cc:2391] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 8: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:59:43.974820 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 111649us
I20250831 07:59:43.976657 31808 raft_consensus.cc:1273] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250831 07:59:43.978303 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:43.986104 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:43.987438 31878 raft_consensus.cc:1238] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:43.988355 31804 raft_consensus.cc:2391] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:43.985544 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:43.993623 31874 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 6.
I20250831 07:59:43.995407 31882 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
I20250831 07:59:43.986732 31877 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
W20250831 07:59:44.002177 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 69378.6us
I20250831 07:59:44.006666 32434 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:44.009613 31732 raft_consensus.cc:1238] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:44.009739 31874 raft_consensus.cc:1273] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 07:59:44.012409 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:44.019580 31728 raft_consensus.cc:1273] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 07:59:43.980494 2508 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:44.022120 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:44.022671 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:44.022773 31801 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:44.024277 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:43.952332 2241 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:43.948912 2504 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:44.025380 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:44.026185 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:44.029685 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:44.026808 31874 raft_consensus.cc:2391] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 6: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:44.002157 2448 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:44.002908 2454 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:44.030730 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:43.951383 2516 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:44.034991 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:44.031463 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 162912us
I20250831 07:59:44.038452 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:44.040436 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:44.040733 31803 raft_consensus.cc:1238] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:44.047926 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:44.048981 31809 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8.
I20250831 07:59:44.051308 32436 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:44.057121 32464 raft_consensus.cc:2366] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 6. Current term is 7.
I20250831 07:59:44.032202 2514 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 07:59:44.059567 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:44.062028 31874 raft_consensus.cc:2391] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 7: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:44.032256 2228 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:44.068619 31809 raft_consensus.cc:1238] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250831 07:59:44.061511 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:44.060494 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:44.070266 31882 raft_consensus.cc:2391] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 in current term 7: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
W20250831 07:59:44.060276 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 289100us
I20250831 07:59:44.060173 32436 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:44.063292 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:44.065016 2241 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:44.065908 31803 raft_consensus.cc:1238] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:44.067057 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:44.034974 2220 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:44.060616 2501 raft_consensus.cc:2747] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:44.072209 31808 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:44.074575 31874 raft_consensus.cc:1273] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 07:59:44.029165 2508 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:44.080281 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:44.075717 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26b32ddce4444eb89ebd7a73a7900bdf" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:44.081771 31877 raft_consensus.cc:2374] T 26b32ddce4444eb89ebd7a73a7900bdf P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7. Re-sending same reply.
I20250831 07:59:44.077970 31880 raft_consensus.cc:2466] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:44.082108 31809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:44.079639 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:44.082881 31809 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Advancing to term 11
I20250831 07:59:44.079615 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:44.085536 31728 raft_consensus.cc:2408] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20250831 07:59:44.086374 31849 leader_election.cc:304] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:44.088338 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:44.089581 31876 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:44.088966 31805 raft_consensus.cc:1238] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250831 07:59:44.061477 2514 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:44.091488 31733 raft_consensus.cc:1273] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 07:59:44.037266 2448 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:44.055858 2471 consensus_queue.cc:1035] T a98f72c022fe465da362553182e4ecca P a0d845d01c8c4894895f66e3beae298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20250831 07:59:44.093645 31874 raft_consensus.cc:1238] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250831 07:59:44.098712 32434 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:44.099059 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:44.099416 31877 raft_consensus.cc:1238] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:44.104383 31876 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 8.
W20250831 07:59:44.101076 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:44.104707 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:44.105284 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:44.105440 31878 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
I20250831 07:59:44.102492 31848 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:44.109632 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:44.108317 31698 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:44.115682 31809 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 11.
I20250831 07:59:44.119535 31728 raft_consensus.cc:1238] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
W20250831 07:59:44.103253 2521 log.cc:927] Time spent T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.126s user 0.000s sys 0.001s
I20250831 07:59:44.044342 2452 consensus_queue.cc:260] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:44.138401 31882 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 5.
I20250831 07:59:44.076929 2275 consensus_queue.cc:1035] T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20250831 07:59:44.098639 2514 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:44.073238 2228 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:44.140554 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 54258.3us
I20250831 07:59:44.079697 1843 consensus_queue.cc:1035] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20250831 07:59:44.110894 2458 raft_consensus.cc:2802] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 07:59:44.098145 2526 raft_consensus.cc:2747] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:44.079720 2220 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:44.170820 31699 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:44.176952 31849 leader_election.cc:304] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:44.141021 2454 raft_consensus.cc:2747] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:44.183522 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:44.183517 31698 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters: 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:44.140328 2286 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:44.198699 31849 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:44.200374 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 97909us
I20250831 07:59:44.203287 31699 leader_election.cc:304] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:44.146796 2527 raft_consensus.cc:2747] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 07:59:44.215406 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 136636us
I20250831 07:59:44.218739 32436 leader_election.cc:304] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:44.143033 2096 consensus_queue.cc:1035] T a62a36682136474ca552fbc53190e09e P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
W20250831 07:59:44.223670 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 119016us
W20250831 07:59:44.215690 2350 consensus_meta.cc:287] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.516s user 0.005s sys 0.006s
I20250831 07:59:44.241948 31775 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:44.264467 32434 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:44.271123 2517 consensus_queue.cc:1046] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20250831 07:59:44.234375 2454 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader election won for term 9
W20250831 07:59:44.288560 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 197245us
I20250831 07:59:44.290352 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:44.291087 31808 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:44.281531 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 98116.3us
W20250831 07:59:44.258714 2520 log.cc:927] Time spent T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.234s user 0.000s sys 0.001s
I20250831 07:59:44.272228 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
W20250831 07:59:44.297978 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 71614.7us
I20250831 07:59:44.243955 2350 raft_consensus.cc:2747] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:44.315120 31800 raft_consensus.cc:2408] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20250831 07:59:44.227348 2474 consensus_queue.cc:1035] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.060s
I20250831 07:59:44.215582 2286 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:44.326508 31876 raft_consensus.cc:1238] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:44.274003 2527 raft_consensus.cc:2747] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250831 07:59:44.325554 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:44.272711 2473 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 07:59:44.343958 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:44.366354 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 71342us
I20250831 07:59:44.366621 31876 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:44.367097 32434 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:44.368788 31878 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 9. Current term is 10. Ops: []
I20250831 07:59:44.369153 31874 raft_consensus.cc:2466] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:44.369390 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f611ea44f17c403e818574844e13b946" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:44.370136 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:44.371004 31874 raft_consensus.cc:2391] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
W20250831 07:59:44.371853 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:44.355026 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:44.286540 2508 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:44.398844 31877 raft_consensus.cc:1238] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: []
I20250831 07:59:44.355989 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:44.235319 2526 consensus_queue.cc:1046] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.081s
I20250831 07:59:44.392683 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:44.395695 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:44.430213 31883 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:44.400941 31805 raft_consensus.cc:2466] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
W20250831 07:59:44.404042 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 205414us
I20250831 07:59:44.440619 31849 leader_election.cc:304] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:44.397331 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:44.313082 2471 raft_consensus.cc:2747] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:44.444769 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:44.316998 2454 raft_consensus.cc:695] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 9 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:44.464309 31848 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
W20250831 07:59:44.452644 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 184219us
I20250831 07:59:44.456023 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:44.271183 2523 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.290s
I20250831 07:59:44.460115 31878 raft_consensus.cc:3058] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Advancing to term 11
I20250831 07:59:44.316216 2516 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:44.292745 2275 consensus_queue.cc:1035] T 3f70249b7dd14c0e804b05303fa06f09 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
I20250831 07:59:44.369019 2504 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:44.477252 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 292800us
W20250831 07:59:44.479561 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 187077us
I20250831 07:59:44.445575 2508 raft_consensus.cc:491] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:44.475646 2302 consensus_meta.cc:287] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.595s user 0.011s sys 0.000s
I20250831 07:59:44.510711 31881 raft_consensus.cc:2391] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 9: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:44.511677 31849 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:44.519354 31698 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:44.479893 2448 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:44.527966 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:44.528831 31730 raft_consensus.cc:2466] T 064c41a8e5dc42e8bdb35225c6aae643 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 8.
I20250831 07:59:44.533609 31775 leader_election.cc:304] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:44.533877 31699 leader_election.cc:304] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:44.541373 31734 raft_consensus.cc:1273] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 07:59:44.518697 2228 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:44.532830 2475 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:44.569725 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 202701us
I20250831 07:59:44.572476 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:44.569288 31847 leader_election.cc:400] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 6. Current term is 7.
I20250831 07:59:44.573851 31847 leader_election.cc:403] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250831 07:59:44.578455 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:44.479444 2530 raft_consensus.cc:2747] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:44.580561 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:44.581256 32469 raft_consensus.cc:2466] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 3.
I20250831 07:59:44.542562 2220 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:44.558842 2506 consensus_meta.cc:287] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.665s user 0.010s sys 0.001s
I20250831 07:59:44.583377 32464 raft_consensus.cc:2391] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:44.593171 31847 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:44.529716 2302 raft_consensus.cc:513] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:44.573957 2526 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:44.530181 2508 raft_consensus.cc:513] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:44.597518 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 78300.2us
I20250831 07:59:44.594421 2542 raft_consensus.cc:2747] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:44.622659 31698 leader_election.cc:304] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:44.573240 2516 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:44.602229 2531 log.cc:927] Time spent T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.337s user 0.001s sys 0.000s
W20250831 07:59:44.603852 2555 log.cc:927] Time spent T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250831 07:59:44.543910 2525 consensus_queue.cc:1046] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.281s
W20250831 07:59:44.603943 2535 log.cc:927] Time spent T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.135s user 0.000s sys 0.001s
W20250831 07:59:44.604264 2534 log.cc:927] Time spent T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.275s user 0.001s sys 0.000s
I20250831 07:59:44.604441 2556 raft_consensus.cc:3053] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Stepping down as leader of term 8
I20250831 07:59:44.604678 2530 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:44.582592 2475 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:44.605671 2544 log.cc:927] Time spent T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.174s user 0.001s sys 0.000s
I20250831 07:59:44.554400 2454 consensus_queue.cc:237] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:44.606777 2547 raft_consensus.cc:2802] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 07:59:44.590590 2545 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:44.609472 2302 leader_election.cc:290] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:44.610141 2335 consensus_queue.cc:1035] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 07:59:44.610527 31807 raft_consensus.cc:2366] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 7. Current term is 8.
I20250831 07:59:44.598765 2506 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:44.611111 2524 raft_consensus.cc:3053] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:44.613698 2543 raft_consensus.cc:2747] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:44.619030 31878 raft_consensus.cc:2466] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 11.
I20250831 07:59:44.624958 31698 leader_election.cc:304] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:44.569057 2541 raft_consensus.cc:2802] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 07:59:44.630878 31808 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
W20250831 07:59:44.658365 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 127026us
I20250831 07:59:44.761936 31849 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:44.682610 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:44.572247 2560 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:44.620784 2526 raft_consensus.cc:491] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:44.630127 2220 raft_consensus.cc:2802] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:44.581756 2536 consensus_queue.cc:1046] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.348s
W20250831 07:59:44.703740 2457 consensus_meta.cc:287] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.891s user 0.012s sys 0.000s
W20250831 07:59:44.735062 31773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.027s sys 0.027s
I20250831 07:59:44.693387 2286 leader_election.cc:290] T 25d573ed55e048098c90bcbb00d12fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:44.700155 2556 raft_consensus.cc:738] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:44.746228 2547 raft_consensus.cc:491] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:44.676647 2516 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:44.747081 2542 raft_consensus.cc:2747] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250831 07:59:44.774042 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 309843us
I20250831 07:59:44.764164 2541 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:44.761412 2524 raft_consensus.cc:738] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:44.690682 2448 consensus_queue.cc:1035] T fcbe3794f3fc40819ffbbfef02a071c3 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
W20250831 07:59:44.786379 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 161484us
I20250831 07:59:44.819375 31698 leader_election.cc:304] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:44.819746 31874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
W20250831 07:59:44.764278 2458 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.004s sys 0.013s
I20250831 07:59:44.762965 2545 raft_consensus.cc:695] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [term 7 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:44.784584 31773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s user 0.028s sys 0.028s
W20250831 07:59:44.822809 31879 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.009s sys 0.034s
W20250831 07:59:44.829425 31879 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.517s user 0.009s sys 0.034s
W20250831 07:59:44.829334 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 761379us
I20250831 07:59:44.772915 2526 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Advancing to term 8
W20250831 07:59:44.852447 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 318624us
I20250831 07:59:44.746589 2530 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:44.826341 2458 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.603s user 0.006s sys 0.014s
I20250831 07:59:44.779749 2220 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:44.851982 31773 leader_election.cc:304] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:44.853619 31881 raft_consensus.cc:1273] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250831 07:59:44.821839 2286 raft_consensus.cc:3058] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:44.811481 2547 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:44.788864 2457 raft_consensus.cc:513] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:44.851804 2542 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:44.822479 2541 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:44.880937 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 119089us
I20250831 07:59:44.807144 2543 consensus_queue.cc:1046] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.084s
W20250831 07:59:44.890686 2533 log.cc:927] Time spent T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.386s user 0.001s sys 0.000s
W20250831 07:59:44.916957 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 97666.5us
I20250831 07:59:44.880417 2561 raft_consensus.cc:2747] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250831 07:59:44.916668 2574 consensus_queue.cc:1046] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20250831 07:59:44.940308 2554 raft_consensus.cc:2802] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 07:59:44.951522 31883 consensus_meta.cc:287] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.507s user 0.005s sys 0.004s
I20250831 07:59:44.779145 2560 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:44.961678 31849 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:44.973752 2586 raft_consensus.cc:2802] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:44.975651 2536 raft_consensus.cc:3053] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:44.903977 2220 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:44.897200 2458 raft_consensus.cc:695] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [term 5 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:44.852499 2350 consensus_queue.cc:1035] T 409f8ff1108b412382695bc6ce64e3cc P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20250831 07:59:44.983929 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:44.880580 2099 consensus_queue.cc:1035] T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.089s
I20250831 07:59:44.985354 31728 raft_consensus.cc:1238] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:44.881019 2335 consensus_queue.cc:1035] T 982febd23d7540aaa3b48b70133e982e P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20250831 07:59:44.997910 31883 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:44.986663 31732 raft_consensus.cc:1238] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:44.911769 2241 raft_consensus.cc:2802] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:45.004381 2554 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:44.979526 2508 leader_election.cc:290] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:45.013911 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 53366.7us
I20250831 07:59:44.941414 2550 consensus_queue.cc:1035] T 953dd19ad1e4414ab1c956437fd8bbfa P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.185s
I20250831 07:59:44.956373 2302 raft_consensus.cc:2747] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:44.978473 2549 raft_consensus.cc:2747] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:44.957294 2542 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.085136 32470 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:45.091712 2506 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:45.091949 2542 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:44.956730 2524 consensus_queue.cc:260] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.092995 2524 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:45.084972 2522 consensus_queue.cc:1035] T 9d3848dba46b4b239a92dc14f93ef48d P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.197s
I20250831 07:59:45.071746 2545 consensus_queue.cc:237] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.071552 2567 raft_consensus.cc:2747] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:45.106361 2585 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:45.068253 2475 leader_election.cc:290] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:45.118175 2581 log.cc:927] Time spent T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
I20250831 07:59:45.118563 2457 leader_election.cc:290] T f7c92dde8fc9490dafb6d51ad8b26855 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:45.118889 2576 raft_consensus.cc:2747] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 07:59:45.119428 2569 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:45.012852 2586 raft_consensus.cc:491] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:45.047542 1966 raft_consensus.cc:3053] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Stepping down as leader of term 8
I20250831 07:59:45.119644 2559 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:45.119760 2530 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:45.133828 2392 consensus_meta.cc:287] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 1.160s user 0.011s sys 0.003s
I20250831 07:59:45.006723 2563 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:45.073716 2554 raft_consensus.cc:3058] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:45.120066 2549 consensus_queue.cc:1046] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20250831 07:59:45.065825 2241 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 07:59:45.131306 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 279407us
I20250831 07:59:45.120440 2588 raft_consensus.cc:491] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:45.127120 2516 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:45.127700 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 166125us
W20250831 07:59:45.191447 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:45.191167 31849 leader_election.cc:304] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:45.017666 2571 consensus_queue.cc:1035] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.150s
W20250831 07:59:45.107110 2551 log.cc:927] Time spent T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.210s user 0.000s sys 0.000s
I20250831 07:59:45.167943 31773 leader_election.cc:304] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:45.136597 1966 raft_consensus.cc:738] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:45.242446 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:45.136196 2586 raft_consensus.cc:3058] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:45.157133 2392 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.143023 2457 consensus_queue.cc:1046] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20250831 07:59:45.171011 2241 raft_consensus.cc:3058] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:45.139420 2458 consensus_queue.cc:237] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.157430 2560 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:45.141769 2559 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.258651 31698 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:45.174826 2529 log.cc:927] Time spent T 14ab15c60aaf42cf911851425eaab94d P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 1.064s user 0.000s sys 0.008s
I20250831 07:59:45.177023 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1546) took 1089 ms. Trace:
I20250831 07:59:45.178252 2573 consensus_queue.cc:1046] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
W20250831 07:59:45.237680 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 185356us
I20250831 07:59:45.131294 2585 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.168555 2567 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:45.255525 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 87680.5us
I20250831 07:59:45.280402 31773 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:45.271112 31879 rpcz_store.cc:276] 0831 07:59:44.087686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.088224 (+ 538us) service_pool.cc:225] Handling call
0831 07:59:44.106643 (+ 18419us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:44.209152 (+102509us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:44.209186 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:44.241397 (+ 32211us) log.cc:844] Serialized 26 byte log entry
0831 07:59:44.829877 (+588480us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:44.890612 (+ 60735us) log.cc:844] Serialized 14 byte log entry
0831 07:59:44.890789 (+ 177us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:44.890852 (+ 63us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:45.176744 (+285892us) raft_consensus.cc:1683] finished
0831 07:59:45.176767 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:45.176929 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10650,"prepare.queue_time_us":60950,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":5648,"raft.queue_time_us":186152,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":68615,"thread_start_us":589078,"threads_started":2,"wal-append.queue_time_us":853891}
W20250831 07:59:45.252092 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 367568us
I20250831 07:59:45.007156 2536 raft_consensus.cc:738] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
W20250831 07:59:45.273460 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:45.253014 2522 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:45.162027 2563 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.289754 2573 consensus_queue.cc:1230] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20250831 07:59:45.376269 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 73617.8us
I20250831 07:59:45.291246 2559 leader_election.cc:290] T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:45.273540 2581 log.cc:927] Time spent T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20250831 07:59:45.162657 2541 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:45.291656 2514 consensus_meta.cc:287] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae: Time spent flushing consensus metadata: real 1.079s user 0.009s sys 0.000s
I20250831 07:59:45.138053 2569 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.270696 2574 consensus_queue.cc:1230] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 07:59:45.287672 2567 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.292945 2392 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:45.291747 2547 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:45.384819 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 104528us
I20250831 07:59:45.385811 31878 raft_consensus.cc:1238] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:45.392866 31699 leader_election.cc:304] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:45.393338 31875 raft_consensus.cc:1273] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250831 07:59:45.394410 31698 leader_election.cc:304] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:45.397532 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:45.292136 2540 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:45.378557 2593 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:45.377104 2474 raft_consensus.cc:2802] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 07:59:45.386787 2350 raft_consensus.cc:2747] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:45.579098 2540 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.586534 2540 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:45.589927 2547 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:45.378638 2577 raft_consensus.cc:3053] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Stepping down as leader of term 7
I20250831 07:59:45.592804 2577 raft_consensus.cc:738] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:45.594339 2543 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:45.594004 2577 consensus_queue.cc:260] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.594904 2543 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.205255 2588 raft_consensus.cc:513] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.556926 31729 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:45.377700 2454 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:45.380875 2513 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:45.503345 31880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1535) took 1384 ms (client timeout 1692 ms). Trace:
I20250831 07:59:45.386404 2563 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:45.391780 2457 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:45.573992 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 181186us
I20250831 07:59:45.636714 31699 leader_election.cc:304] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:45.598165 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:45.615432 31880 rpcz_store.cc:269] 0831 07:59:44.009654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.012265 (+ 2611us) service_pool.cc:225] Handling call
0831 07:59:45.392813 (+1380548us) spinlock_profiling.cc:243] Waited 52.9 ms on lock 0x7b5400299ad4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:45.394164 (+ 1351us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":121659776}
I20250831 07:59:45.637789 31878 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 9.
W20250831 07:59:45.594357 2524 consensus_meta.cc:287] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.501s user 0.008s sys 0.000s
I20250831 07:59:45.651297 2513 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.656656 2457 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.661949 2506 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:45.379694 1966 consensus_queue.cc:260] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.576570 2574 consensus_queue.cc:1046] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.104s
W20250831 07:59:45.663415 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:45.664234 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 473144us
I20250831 07:59:45.293350 2286 raft_consensus.cc:2747] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319) with higher term. Message: Invalid argument: T 59e33bfa463944f89cfbb00d1247323a P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 6. Current term is 7.
I20250831 07:59:45.665647 2569 leader_election.cc:290] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:45.670699 31699 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:45.595659 2577 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:45.482513 2573 consensus_queue.cc:1046] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.153s
I20250831 07:59:45.595999 2561 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250831 07:59:45.616217 2555 log.cc:927] Time spent T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.100s user 0.000s sys 0.001s
W20250831 07:59:45.631345 2544 log.cc:927] Time spent T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
I20250831 07:59:45.689203 31729 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 10.
I20250831 07:59:45.382903 2526 raft_consensus.cc:513] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.754547 32462 raft_consensus.cc:1238] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8. Ops: []
I20250831 07:59:45.754730 2526 leader_election.cc:290] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:45.756448 2585 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:45.758082 2563 raft_consensus.cc:2802] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 11 FOLLOWER]: Leader election won for term 11
I20250831 07:59:45.758563 2563 raft_consensus.cc:695] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [term 11 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
W20250831 07:59:45.759081 2554 consensus_meta.cc:287] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.568s user 0.008s sys 0.001s
I20250831 07:59:45.661955 2542 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:45.697549 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:45.802901 32469 raft_consensus.cc:2466] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 9.
I20250831 07:59:45.698369 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:45.596918 2591 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:45.663097 2543 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:45.716707 31849 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:45.814222 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 97597.1us
I20250831 07:59:45.815181 31849 leader_election.cc:304] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:45.817104 31849 leader_election.cc:304] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:45.818445 31849 leader_election.cc:304] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:45.664784 2516 raft_consensus.cc:2802] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:45.721161 2598 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:45.722859 2513 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:45.722980 2550 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:45.722872 2568 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:45.723640 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:45.724459 2457 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:45.724699 2541 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 5 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:45.712373 2506 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.730084 32465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:45.730484 2471 raft_consensus.cc:2747] T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 07:59:45.730859 1966 raft_consensus.cc:3058] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:45.515492 2514 raft_consensus.cc:513] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } attrs { replace: true } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:45.746485 31699 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 75892.6us
I20250831 07:59:45.759347 31874 raft_consensus.cc:2466] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7.
I20250831 07:59:45.632845 31773 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:45.872737 31699 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:45.803957 32463 raft_consensus.cc:3058] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:45.804625 31847 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:45.689962 2522 consensus_queue.cc:1035] T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.061s
I20250831 07:59:45.825984 2286 raft_consensus.cc:3058] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:45.738459 2567 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:45.749219 2569 raft_consensus.cc:2747] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:45.893796 2569 raft_consensus.cc:2747] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20250831 07:59:45.894735 31699 connection.cc:232] RPC call timeout handler was delayed by -1.4469429399999854s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20250831 07:59:45.895344 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:45.897214 2601 log.cc:927] Time spent T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.314s user 0.000s sys 0.001s
I20250831 07:59:45.897635 2506 leader_election.cc:290] T 518e6f5a91554de78e0bd0b6ec04c12a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:45.829314 2516 raft_consensus.cc:695] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
W20250831 07:59:45.826346 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 432439us
I20250831 07:59:45.796942 2573 consensus_queue.cc:1230] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 7512ba2ed3a94982a22a75f51a647e65 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250831 07:59:45.910404 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1532) took 1901 ms. Trace:
I20250831 07:59:45.910768 31875 rpcz_store.cc:276] 0831 07:59:44.008741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.021442 (+ 12701us) service_pool.cc:225] Handling call
0831 07:59:45.392979 (+1371537us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:45.556330 (+163351us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:45.910227 (+353897us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400299ad0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:45.910356 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3121068800}
I20250831 07:59:45.807641 2588 leader_election.cc:290] T a62a36682136474ca552fbc53190e09e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:45.914426 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:45.819612 2561 raft_consensus.cc:2747] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20250831 07:59:45.872798 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 240212us
I20250831 07:59:45.914660 2556 consensus_queue.cc:260] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:45.798102 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 166968us
I20250831 07:59:45.916244 2602 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:45.916795 2602 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.918769 2602 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:45.919360 31881 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:45.920096 31881 raft_consensus.cc:2391] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
W20250831 07:59:45.874573 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 70052.9us
I20250831 07:59:45.921183 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:45.921727 31883 raft_consensus.cc:3058] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:45.922714 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:45.821663 2454 raft_consensus.cc:695] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [term 7 LEADER]: Becoming Leader. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:45.828245 2543 raft_consensus.cc:2747] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 07:59:45.631628 2593 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.830611 2513 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 07:59:45.883986 2558 raft_consensus.cc:2802] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:45.891630 2457 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:45.812721 2542 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.831264 2550 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.759344 2563 consensus_queue.cc:237] T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 9.4, Last appended by leader: 4, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.804924 2554 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.911752 2588 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:45.901535 2514 leader_election.cc:290] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:45.865859 2526 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:45.667793 2536 consensus_queue.cc:260] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.882644 2541 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 07:59:45.904908 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1921) took 1223 ms. Trace:
I20250831 07:59:45.905095 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:45.826479 2591 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.827481 2598 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.823676 31849 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:45.907501 2584 raft_consensus.cc:2747] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 07:59:45.928303 32434 leader_election.cc:304] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:45.929483 31773 leader_election.cc:304] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:45.933036 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:45.934128 32463 raft_consensus.cc:2466] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:45.920173 32465 raft_consensus.cc:2391] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 5: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:45.942555 31883 raft_consensus.cc:2466] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 8.
I20250831 07:59:45.931392 2392 consensus_queue.cc:1046] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20250831 07:59:45.900483 31729 raft_consensus.cc:1273] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250831 07:59:46.032778 31881 raft_consensus.cc:1238] T 9a40df3e72bd462091ca307d230a90ed P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 8. Ops: []
W20250831 07:59:46.028743 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 205095us
I20250831 07:59:45.953768 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "518e6f5a91554de78e0bd0b6ec04c12a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:45.957935 31878 raft_consensus.cc:3058] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Advancing to term 5
W20250831 07:59:45.937269 2220 consensus_meta.cc:287] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.888s user 0.012s sys 0.000s
I20250831 07:59:45.937031 2558 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:45.976835 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a40df3e72bd462091ca307d230a90ed" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:46.040277 31727 raft_consensus.cc:1238] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 8. Current term is 9. Ops: []
I20250831 07:59:46.041272 31727 raft_consensus.cc:1238] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 8. Current term is 9. Ops: []
I20250831 07:59:45.976409 2541 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:46.046707 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:46.048609 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 119214us
I20250831 07:59:46.050940 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:45.855360 2568 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:45.976883 31880 raft_consensus.cc:1238] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [term 4 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 3. Current term is 4. Ops: []
I20250831 07:59:46.021083 2557 raft_consensus.cc:491] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.065253 31878 raft_consensus.cc:2466] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:45.976298 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:45.943717 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:45.949371 31875 raft_consensus.cc:1273] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Refusing update from remote peer 7ecb257174554a26b2c35ad04b0eabae: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250831 07:59:46.033654 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:46.066044 31877 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:45.935211 2099 consensus_queue.cc:1046] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
W20250831 07:59:46.034176 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 426418us
I20250831 07:59:45.970135 2513 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:46.012784 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:46.013417 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:46.007622 2608 log.cc:927] Time spent T fc393e9ddc4c42dfa208cc7251b0b15b P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.130s user 0.000s sys 0.001s
I20250831 07:59:46.038177 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:46.032964 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 104766us
I20250831 07:59:45.970078 31882 rpcz_store.cc:276] 0831 07:59:44.681948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.682228 (+ 280us) service_pool.cc:225] Handling call
0831 07:59:44.758767 (+ 76539us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:45.238631 (+479864us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:45.238678 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:45.239266 (+ 588us) log.cc:844] Serialized 26 byte log entry
0831 07:59:45.391515 (+152249us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:45.391596 (+ 81us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:45.391661 (+ 65us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:45.904711 (+513050us) raft_consensus.cc:1683] finished
0831 07:59:45.904730 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:45.904855 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":539998,"raft.queue_time_us":290941,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":48273,"thread_start_us":631120,"threads_started":2,"wal-append.queue_time_us":507601}
I20250831 07:59:46.043220 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:46.076254 31729 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 9: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:46.077193 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:45.943411 2567 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:46.041889 2558 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:46.052284 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:46.053305 2518 raft_consensus.cc:2802] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:46.048242 2543 consensus_queue.cc:1035] T 064c41a8e5dc42e8bdb35225c6aae643 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
W20250831 07:59:46.054265 2241 consensus_meta.cc:287] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.753s user 0.007s sys 0.002s
I20250831 07:59:45.977001 2590 consensus_queue.cc:1046] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.078s
I20250831 07:59:46.054431 2591 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:46.055094 2542 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:46.055768 2593 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:46.056259 2392 raft_consensus.cc:3053] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:46.056366 2579 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.055781 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:46.058286 2598 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:46.059015 2609 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.059603 2506 raft_consensus.cc:491] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.060951 2550 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:46.045529 2220 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.066999 32464 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 9: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:46.007390 2454 consensus_queue.cc:237] T e93b827594384bcf80c7487b061b4c24 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.037076 2458 raft_consensus.cc:2747] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:45.957768 2457 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.041468 2585 raft_consensus.cc:2802] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 07:59:46.050088 2541 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:46.091926 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.129665 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.092092 31876 raft_consensus.cc:1238] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 LEADER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250831 07:59:46.093231 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f9ace15ae4422682ce241dda87dedf" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:45.965121 2588 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:45.970793 2526 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.077872 2557 raft_consensus.cc:513] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.091411 2513 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.175289 31727 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
I20250831 07:59:46.122932 2609 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.085947 2599 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.086416 2518 raft_consensus.cc:491] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:46.123085 31775 leader_election.cc:304] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:46.198128 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:46.104787 2392 raft_consensus.cc:738] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:46.117007 2579 raft_consensus.cc:513] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.201678 2553 raft_consensus.cc:3053] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:46.130650 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:46.178159 32434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 99607.5us
I20250831 07:59:46.203568 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:46.204501 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1610) took 2093 ms. Trace:
I20250831 07:59:46.233788 31876 raft_consensus.cc:2466] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:46.233842 32434 leader_election.cc:304] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:46.234838 31876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:46.194185 2474 raft_consensus.cc:491] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:46.126822 2585 raft_consensus.cc:491] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:46.112026 2522 consensus_queue.cc:1035] T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
W20250831 07:59:46.207031 2619 log.cc:927] Time spent T 26b32ddce4444eb89ebd7a73a7900bdf P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
W20250831 07:59:46.207556 2596 log.cc:927] Time spent T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.543s user 0.010s sys 0.004s
I20250831 07:59:46.208343 32469 raft_consensus.cc:1238] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7. Ops: []
W20250831 07:59:46.208819 2594 log.cc:927] Time spent T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.621s user 0.001s sys 0.008s
I20250831 07:59:46.208997 2574 consensus_queue.cc:1046] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 07:59:46.210178 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d055e59fc97471a93e491ee81afa361" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.098440 2241 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.116767 31877 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 9.
I20250831 07:59:46.181156 2554 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 4 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:46.103154 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 89892.7us
I20250831 07:59:46.101938 2567 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.214115 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:46.248260 31877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:46.248574 31879 raft_consensus.cc:3058] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
W20250831 07:59:46.248744 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 60594.4us
I20250831 07:59:46.131515 31883 raft_consensus.cc:2466] T ccd51d2e599e4535ad0f2f6afcb1bfee P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:46.041374 2516 consensus_queue.cc:237] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.221329 2568 leader_election.cc:290] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
W20250831 07:59:46.227680 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:46.131090 2591 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.120895 2506 raft_consensus.cc:513] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.228639 2518 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Advancing to term 6
W20250831 07:59:46.226125 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 103089us
I20250831 07:59:46.253937 31877 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20250831 07:59:46.237310 2585 raft_consensus.cc:3058] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:46.236816 2099 raft_consensus.cc:2802] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Leader election won for term 10
I20250831 07:59:46.236275 31876 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 9.
I20250831 07:59:46.274989 2591 raft_consensus.cc:513] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.233783 31733 rpcz_store.cc:276] 0831 07:59:44.111241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.111400 (+ 159us) service_pool.cc:225] Handling call
0831 07:59:44.139235 (+ 27835us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:44.215217 (+ 75982us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:44.215262 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:44.215333 (+ 71us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:44.215363 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:46.204218 (+1988855us) spinlock_profiling.cc:243] Waited 75.7 ms on lock 0x7b54001a8e54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:46.204267 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:46.204426 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174212992}
I20250831 07:59:46.237638 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.228811 2599 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.280162 2513 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:46.280674 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.280865 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.281466 31733 raft_consensus.cc:2408] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20250831 07:59:46.281646 2599 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:46.281812 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.282172 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:46.127760 2593 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:46.242796 32471 raft_consensus.cc:2466] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:46.251856 2574 consensus_queue.cc:1230] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20250831 07:59:46.232491 2586 consensus_meta.cc:287] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c: Time spent flushing consensus metadata: real 0.958s user 0.006s sys 0.005s
I20250831 07:59:46.188254 2457 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:46.252250 31698 leader_election.cc:304] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:46.204806 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:46.274222 31877 raft_consensus.cc:1238] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:46.274044 2541 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.298830 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.237979 2474 raft_consensus.cc:3058] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:46.232662 2553 raft_consensus.cc:738] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:46.244303 2514 raft_consensus.cc:2802] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 8 FOLLOWER]: Leader election won for term 8
W20250831 07:59:46.275655 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:46.315258 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:46.277976 31729 raft_consensus.cc:2391] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 7: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:46.282873 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.139123 30564 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 07:59:46.246105 31731 raft_consensus.cc:2466] T 4d055e59fc97471a93e491ee81afa361 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 9.
I20250831 07:59:46.298193 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1611) took 2121 ms. Trace:
I20250831 07:59:46.308543 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:46.308512 31882 raft_consensus.cc:1238] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 9. Current term is 10. Ops: []
I20250831 07:59:46.246879 1966 consensus_queue.cc:1035] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
W20250831 07:59:46.313349 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 61167.9us
I20250831 07:59:46.318485 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1630) took 1335 ms. Trace:
I20250831 07:59:46.319000 31731 rpcz_store.cc:276] 0831 07:59:44.982642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.983063 (+ 421us) service_pool.cc:225] Handling call
0831 07:59:46.210019 (+1226956us) spinlock_profiling.cc:243] Waited 2.88 ms on lock 0x7b5800000f98. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e102 00007fdcedd59c39 00007fdcecee5000 00007fdcecfabeab 00007fdcecfabf31 00007fdcecf5ab50 00007fdcecf5a434 00007fdce65e90c4 00007fdce65e9052
0831 07:59:46.318405 (+108386us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6627712}
I20250831 07:59:46.316525 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1623) took 1696 ms. Trace:
I20250831 07:59:46.320093 31725 rpcz_store.cc:276] 0831 07:59:44.620142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.620261 (+ 119us) service_pool.cc:225] Handling call
0831 07:59:44.620653 (+ 392us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:44.620965 (+ 312us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:44.620992 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:44.690429 (+ 69437us) log.cc:844] Serialized 26 byte log entry
0831 07:59:45.251144 (+560715us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:45.251205 (+ 61us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:45.251307 (+ 102us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:46.208172 (+956865us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:46.316179 (+108007us) raft_consensus.cc:1683] finished
0831 07:59:46.316228 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:46.316456 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":200682,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":60492,"spinlock_wait_cycles":356410624,"thread_start_us":472695,"threads_started":1,"wal-append.queue_time_us":830726}
I20250831 07:59:46.322193 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc393e9ddc4c42dfa208cc7251b0b15b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.322286 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e3be3f093d144268837fd81ceb8b44a" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.322945 31725 raft_consensus.cc:2408] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250831 07:59:46.323251 31731 raft_consensus.cc:2391] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 7: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:46.320039 31728 rpcz_store.cc:276] 0831 07:59:44.167182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.167363 (+ 181us) service_pool.cc:225] Handling call
0831 07:59:44.205134 (+ 37771us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:45.047423 (+842289us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:46.288127 (+1240704us) spinlock_profiling.cc:243] Waited 780 ms on lock 0x7b5400090e54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:46.288308 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1793848576}
I20250831 07:59:46.282968 2302 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.326578 31734 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.331041 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.323382 31879 raft_consensus.cc:2466] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 8.
I20250831 07:59:46.345244 31725 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 9: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:46.346486 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:46.347229 31879 raft_consensus.cc:2466] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:46.278466 2584 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.353952 2624 raft_consensus.cc:491] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.354570 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.355473 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:46.349792 2623 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.356706 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9fcfb2ddfa4c6eb4e9264f55b5dba0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:46.356464 2623 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.358664 2623 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:46.299284 2586 raft_consensus.cc:513] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.281652 2567 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:46.360853 2586 leader_election.cc:290] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:46.325569 2457 raft_consensus.cc:2747] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:46.345969 2506 leader_election.cc:290] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:46.362540 31732 raft_consensus.cc:1238] T 838ecfd85da44202b231fff2105cde0b P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:46.363761 31732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1631) took 1201 ms. Trace:
I20250831 07:59:46.364114 31732 rpcz_store.cc:276] 0831 07:59:45.162239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:45.168992 (+ 6753us) service_pool.cc:225] Handling call
0831 07:59:45.250465 (+ 81473us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:46.362940 (+1112475us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:46.363520 (+ 580us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54001a9354. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:46.363707 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2557074304}
I20250831 07:59:46.348304 2099 raft_consensus.cc:695] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [term 10 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:46.350562 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:46.367093 2099 consensus_queue.cc:237] T 90effb264f51485dac220ee32f49e4ad P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.369026 2541 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 10 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:46.369678 2541 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.370091 2541 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.370765 31734 raft_consensus.cc:1273] T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 07:59:46.372143 2220 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:46.372581 2526 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:46.348717 2625 raft_consensus.cc:491] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:46.354872 2574 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:46.377982 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91390ee2f1e2445ca47e8cb7bd8eeecd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:46.350667 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eb1971f5bb14c27bdbad34649f50ab4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:46.384552 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 61567.2us
I20250831 07:59:46.353379 31849 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:46.388345 31879 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:46.388453 31698 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:46.390118 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:46.392524 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1620) took 1864 ms. Trace:
I20250831 07:59:46.400046 31698 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:46.402573 31698 leader_election.cc:304] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:46.404035 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:46.407035 31733 raft_consensus.cc:2466] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 7.
I20250831 07:59:46.297344 2593 raft_consensus.cc:513] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.407662 31883 raft_consensus.cc:1238] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:46.352669 2241 leader_election.cc:290] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:46.455799 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:46.450577 2532 consensus_queue.cc:1046] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
W20250831 07:59:46.458051 2600 log.cc:927] Time spent T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.796s user 0.009s sys 0.000s
I20250831 07:59:46.393013 31730 rpcz_store.cc:276] 0831 07:59:44.528166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.529658 (+ 1492us) service_pool.cc:225] Handling call
0831 07:59:44.529962 (+ 304us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:44.745381 (+215419us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 07:59:44.771712 (+ 26331us) log.cc:844] Serialized 14 byte log entry
0831 07:59:45.252143 (+480431us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:45.255428 (+ 3285us) log.cc:844] Serialized 26 byte log entry
0831 07:59:45.255584 (+ 156us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:45.255620 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:45.255699 (+ 79us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:46.206883 (+951184us) spinlock_profiling.cc:243] Waited 80.1 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:46.373148 (+166265us) raft_consensus.cc:1683] finished
0831 07:59:46.392286 (+ 19138us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:46.392456 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36218,"prepare.queue_time_us":38199,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":186216,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":45436,"spinlock_wait_cycles":184115456,"thread_start_us":586035,"threads_started":2,"wal-append.queue_time_us":669911}
W20250831 07:59:46.412083 31807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1942) took 2345 ms (client timeout 1654 ms). Trace:
W20250831 07:59:46.461144 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 107897us
W20250831 07:59:46.461200 31807 rpcz_store.cc:269] 0831 07:59:44.056661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.056843 (+ 182us) service_pool.cc:225] Handling call
0831 07:59:46.401679 (+2344836us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b54000e97d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:46.401826 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1238914432}
I20250831 07:59:46.462428 31849 leader_election.cc:304] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:46.353396 2302 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.426584 31698 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:46.414134 31875 raft_consensus.cc:2391] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:46.354517 2624 raft_consensus.cc:513] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:46.472141 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 45703.9us
I20250831 07:59:46.472625 31883 raft_consensus.cc:1273] T 3eb1971f5bb14c27bdbad34649f50ab4 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 07:59:46.477700 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "838ecfd85da44202b231fff2105cde0b" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:46.397742 2586 raft_consensus.cc:2802] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:46.391692 2541 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:46.458465 2454 raft_consensus.cc:3053] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 6 LEADER]: Stepping down as leader of term 6
I20250831 07:59:46.509495 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:46.412343 2599 raft_consensus.cc:2802] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:46.411909 2506 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 8247a38b550649aa9cd6e7668dcaec10 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 07:59:46.556296 31883 raft_consensus.cc:2391] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 5: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:46.557359 31728 raft_consensus.cc:2408] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250831 07:59:46.492503 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:46.408270 2513 raft_consensus.cc:2802] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:46.408043 2567 consensus_queue.cc:1046] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.151s
I20250831 07:59:46.560344 31875 raft_consensus.cc:1273] T 2068a5cd20ba438a9dae813ac5acda03 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20250831 07:59:46.556674 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 94399us
I20250831 07:59:46.561093 2501 raft_consensus.cc:491] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d845d01c8c4894895f66e3beae298c)
I20250831 07:59:46.564076 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:46.564774 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:46.565210 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:46.594314 31803 raft_consensus.cc:3058] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Advancing to term 9
I20250831 07:59:46.594969 31808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:46.565315 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:46.565551 2579 leader_election.cc:290] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:46.567278 31809 raft_consensus.cc:1238] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Rejecting Update request from peer 7ecb257174554a26b2c35ad04b0eabae for earlier term 7. Current term is 8. Ops: []
I20250831 07:59:46.457086 2595 raft_consensus.cc:3053] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 07:59:46.484982 2099 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.465602 2543 raft_consensus.cc:3058] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:46.572032 31802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:46.492378 2586 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:46.574101 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:46.570101 2599 raft_consensus.cc:491] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:46.578848 31849 leader_election.cc:304] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
I20250831 07:59:46.583209 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:46.385172 2584 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.456496 2588 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:46.485282 2532 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250831 07:59:46.605576 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:46.596972 2567 consensus_queue.cc:1230] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer 8247a38b550649aa9cd6e7668dcaec10 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250831 07:59:46.571359 32468 raft_consensus.cc:3058] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:46.555787 2454 raft_consensus.cc:738] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:46.648342 31698 leader_election.cc:304] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:46.615470 31875 raft_consensus.cc:2391] T e3a3b559f26e43f8942b8753f3e30607 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 7: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:46.562875 31805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:46.600446 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b2e336f43934f118ecacdc1e1d2da60" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:46.619150 31883 raft_consensus.cc:2466] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 5.
I20250831 07:59:46.403594 2625 raft_consensus.cc:513] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.572921 2513 raft_consensus.cc:491] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:46.623392 31809 raft_consensus.cc:1273] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250831 07:59:46.595968 2448 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.596168 2554 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.484766 2514 raft_consensus.cc:695] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [term 8 LEADER]: Becoming Leader. State: Replica: 7ecb257174554a26b2c35ad04b0eabae, State: Running, Role: LEADER
I20250831 07:59:46.608978 2595 raft_consensus.cc:738] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:46.630314 2632 raft_consensus.cc:491] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:46.617698 2586 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:46.634610 2518 raft_consensus.cc:513] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.635217 31773 leader_election.cc:304] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:46.635915 2630 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:46.636178 2532 raft_consensus.cc:491] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:46.636952 2584 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:46.612872 2591 leader_election.cc:290] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:46.602203 2220 consensus_queue.cc:1035] T 54f9ace15ae4422682ce241dda87dedf P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20250831 07:59:46.446027 2553 consensus_queue.cc:260] T 9e3be3f093d144268837fd81ceb8b44a P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.618197 2099 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.639788 2506 raft_consensus.cc:2802] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 07:59:46.639984 2633 raft_consensus.cc:491] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:46.627597 2627 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.641376 2593 leader_election.cc:290] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:46.642346 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e33bfa463944f89cfbb00d1247323a" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:46.645666 2536 raft_consensus.cc:3058] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:46.408380 2574 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.637312 2618 raft_consensus.cc:491] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.603248 2501 raft_consensus.cc:513] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.730711 2585 raft_consensus.cc:513] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:46.837880 2286 consensus_meta.cc:287] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10: Time spent flushing consensus metadata: real 0.922s user 0.010s sys 0.002s
I20250831 07:59:46.844969 32467 raft_consensus.cc:1238] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 10 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 9. Current term is 10. Ops: []
I20250831 07:59:46.846055 2286 raft_consensus.cc:2747] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 07:59:46.846120 31883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:46.656246 2588 raft_consensus.cc:491] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:46.848152 2585 leader_election.cc:290] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 10 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:46.707469 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:46.693208 31806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39082 (request call id 1943) took 2582 ms. Trace:
W20250831 07:59:46.781102 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:46.795279 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 216492us
I20250831 07:59:46.844030 31805 raft_consensus.cc:2374] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 7. Re-sending same reply.
I20250831 07:59:46.846880 31883 raft_consensus.cc:2466] T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 6.
I20250831 07:59:46.851876 31806 rpcz_store.cc:276] 0831 07:59:44.075797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.075961 (+ 164us) service_pool.cc:225] Handling call
0831 07:59:44.087261 (+ 11300us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:44.213065 (+125804us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:44.213112 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:44.613065 (+399953us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b18002e5f40. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdcedd40e31 00007fdcedd4d781 00007fdce509b792 00007fdce509db61 00007fdce509db19 00007fdce509daa9 00007fdce509da71 00007fdce509cd6d 0000000000379254 0000000000379099 00007fdce507eb1a
0831 07:59:45.160861 (+547796us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x7b2c00000c88. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce69a09a1 00007fdce69affd2 00007fdce8039021 00007fdce7ffd3fa 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32
0831 07:59:45.178447 (+ 17586us) log.cc:844] Serialized 26 byte log entry
0831 07:59:45.299689 (+121242us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:45.299731 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:46.658082 (+1358351us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b5400235d54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:46.658149 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:46.658215 (+ 66us) raft_consensus.cc:1683] finished
0831 07:59:46.658237 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:46.658419 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16880,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"spinlock_wait_cycles":285110144,"thread_start_us":120837,"threads_started":1,"wal-append.queue_time_us":723494}
I20250831 07:59:46.621672 2599 raft_consensus.cc:3058] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:46.872670 31875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:46.871790 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 236478us
W20250831 07:59:46.875274 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:46.879742 31875 raft_consensus.cc:2466] T e93b827594384bcf80c7487b061b4c24 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
W20250831 07:59:46.886358 31775 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:46.848225 2574 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:46.853919 2513 raft_consensus.cc:3058] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:46.690900 2457 raft_consensus.cc:2802] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:46.893712 31808 raft_consensus.cc:2391] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:46.860110 2532 raft_consensus.cc:3058] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:46.894402 31773 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters: 7ecb257174554a26b2c35ad04b0eabae
I20250831 07:59:46.715641 2563 raft_consensus.cc:491] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:46.870707 2543 raft_consensus.cc:2747] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:46.870154 2624 leader_election.cc:290] T db51a15157664ec0a9d8402ca441f042 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:46.872516 2593 raft_consensus.cc:491] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:46.907116 32468 raft_consensus.cc:2466] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 10.
I20250831 07:59:46.907140 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:46.908731 31803 raft_consensus.cc:2466] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 9.
I20250831 07:59:46.717763 2241 consensus_queue.cc:1046] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.178s
I20250831 07:59:46.915573 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2068a5cd20ba438a9dae813ac5acda03" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:46.918270 31806 raft_consensus.cc:2391] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 11: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:46.897130 2457 raft_consensus.cc:491] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:46.625492 2474 raft_consensus.cc:513] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.848198 2588 raft_consensus.cc:513] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.894490 2302 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:46.868543 2554 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:46.868746 2448 raft_consensus.cc:513] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.892956 2635 raft_consensus.cc:2802] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20250831 07:59:46.918644 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 243140us
W20250831 07:59:46.948534 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 46416.8us
I20250831 07:59:46.949455 32434 leader_election.cc:304] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:46.914368 2335 raft_consensus.cc:2802] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 07:59:46.903379 2563 raft_consensus.cc:513] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.919490 2593 raft_consensus.cc:513] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.913368 2595 consensus_queue.cc:260] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.843920 2632 raft_consensus.cc:513] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.858610 2618 raft_consensus.cc:513] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:46.867496 2630 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:46.846882 2627 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.848387 2633 raft_consensus.cc:513] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:46.918503 2454 consensus_queue.cc:260] T fe2bab0189fe475cb540bdf7a041d8b8 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:47.050772 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054807582f54efba5d989f569ab0666" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:47.159253 2625 leader_election.cc:290] T 000f9ab653554f498fe14065e6656a62 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
W20250831 07:59:46.978677 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:47.024775 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:47.079384 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3bdbd8499374432a94ff0909a4c5f95" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:47.079340 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9560a731dd294178911faa26dbfc2ef8" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:47.008002 31804 raft_consensus.cc:1238] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 7. Current term is 9. Ops: []
I20250831 07:59:47.009294 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
W20250831 07:59:46.997020 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 47198.8us
I20250831 07:59:46.941406 2457 raft_consensus.cc:3058] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:47.115242 31698 leader_election.cc:304] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:47.057736 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:47.169184 31879 raft_consensus.cc:2391] T 4f319de3ac3c4b9a9b87cd497e02c51c P 8247a38b550649aa9cd6e7668dcaec10 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 10: Already voted for candidate 8247a38b550649aa9cd6e7668dcaec10 in this term.
I20250831 07:59:46.945622 2584 raft_consensus.cc:491] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:47.013760 2335 raft_consensus.cc:695] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 LEADER]: Becoming Leader. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:47.167990 31728 raft_consensus.cc:2466] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 6.
W20250831 07:59:47.091223 2638 log.cc:927] Time spent T 25d573ed55e048098c90bcbb00d12fdb P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20250831 07:59:47.008329 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:47.072707 31849 leader_election.cc:304] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters: 7512ba2ed3a94982a22a75f51a647e65
W20250831 07:59:47.097429 2639 log.cc:927] Time spent T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250831 07:59:46.990708 2574 raft_consensus.cc:3053] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 9 LEADER]: Stepping down as leader of term 9
W20250831 07:59:47.164644 2558 consensus_meta.cc:287] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 1.062s user 0.011s sys 0.001s
I20250831 07:59:46.955410 2635 raft_consensus.cc:491] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:47.179587 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
W20250831 07:59:47.179913 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 107295us
I20250831 07:59:47.180593 31805 raft_consensus.cc:1238] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 9. Current term is 10. Ops: []
W20250831 07:59:47.180177 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.3:42585, user_credentials={real_user=slave}} blocked reactor thread for 65067.5us
I20250831 07:59:47.044888 2640 raft_consensus.cc:491] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:46.973004 2516 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:47.004554 2623 raft_consensus.cc:2802] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 07:59:46.946211 2241 raft_consensus.cc:491] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:47.182511 31730 raft_consensus.cc:2408] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20250831 07:59:46.943527 2506 raft_consensus.cc:695] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [term 7 LEADER]: Becoming Leader. State: Replica: 8247a38b550649aa9cd6e7668dcaec10, State: Running, Role: LEADER
I20250831 07:59:46.985502 2096 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:47.054112 2099 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:47.024775 2474 leader_election.cc:290] T 064c41a8e5dc42e8bdb35225c6aae643 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:47.091202 2641 log.cc:927] Time spent T 953dd19ad1e4414ab1c956437fd8bbfa P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20250831 07:59:47.094187 2501 leader_election.cc:290] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:47.023639 2518 leader_election.cc:290] T a98f72c022fe465da362553182e4ecca P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:47.178885 2610 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:47.218945 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:47.199257 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd51d2e599e4535ad0f2f6afcb1bfee" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:47.190732 31773 leader_election.cc:304] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10; no voters:
I20250831 07:59:47.197140 2629 consensus_queue.cc:1035] T 2068a5cd20ba438a9dae813ac5acda03 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20250831 07:59:47.200876 2448 leader_election.cc:290] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:47.188256 2651 raft_consensus.cc:2802] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:47.205201 2506 consensus_queue.cc:237] T 91390ee2f1e2445ca47e8cb7bd8eeecd P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:47.209035 2586 raft_consensus.cc:513] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:47.210506 2599 raft_consensus.cc:513] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:47.199368 2558 raft_consensus.cc:513] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:47.213542 31849 leader_election.cc:304] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:47.196177 31803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2acd7b3ac5f94f53ac1ef93264415686" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:47.220634 31698 leader_election.cc:304] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d845d01c8c4894895f66e3beae298c; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
I20250831 07:59:47.222003 31806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b606cb7b5ccc4e2588ca98715c021620" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:47.222200 31733 raft_consensus.cc:3053] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:47.222868 31732 raft_consensus.cc:1273] T 660e7b2f56414f4fbd04a1929c99b390 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 07:59:47.237174 31733 raft_consensus.cc:738] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:47.191673 2647 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:47.250718 2652 consensus_queue.cc:1035] T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 07:59:47.252140 31800 raft_consensus.cc:2391] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ecb257174554a26b2c35ad04b0eabae in current term 6: Already voted for candidate 7512ba2ed3a94982a22a75f51a647e65 in this term.
I20250831 07:59:47.252462 31733 consensus_queue.cc:260] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:47.253516 31733 raft_consensus.cc:3058] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:47.254835 2457 raft_consensus.cc:513] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:47.255688 2335 consensus_queue.cc:237] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:47.257529 2558 leader_election.cc:290] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 election: Requested vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:47.258477 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:47.260211 32433 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:47.186720 2584 raft_consensus.cc:513] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:47.263774 2584 leader_election.cc:290] T 26525016efb247c3b1c2bf7fec0e1658 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:47.263159 2627 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:47.266738 2549 raft_consensus.cc:2747] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250831 07:59:47.268193 2647 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:47.270589 2647 leader_election.cc:290] T 855219eabb9f4417ad63ec7efa7982da P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:47.272150 31733 raft_consensus.cc:1273] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Refusing update from remote peer 8247a38b550649aa9cd6e7668dcaec10: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20250831 07:59:47.273963 31733 pending_rounds.cc:85] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c: Aborting all ops after (but not including) 2
I20250831 07:59:47.274629 31733 pending_rounds.cc:107] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20250831 07:59:47.275640 31733 raft_consensus.cc:2887] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 07:59:47.188418 2588 leader_election.cc:290] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:47.276844 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93b827594384bcf80c7487b061b4c24" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:47.277546 2575 raft_consensus.cc:3053] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 9 LEADER]: Stepping down as leader of term 9
I20250831 07:59:47.278040 2575 raft_consensus.cc:738] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:47.278227 31732 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:47.278299 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:47.278939 2575 consensus_queue.cc:260] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } }
I20250831 07:59:47.280031 31733 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:47.197598 2635 raft_consensus.cc:3058] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:47.282472 32462 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38366 (request call id 553) took 2275 ms. Trace:
I20250831 07:59:47.282905 32462 rpcz_store.cc:276] 0831 07:59:45.007040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:45.014250 (+ 7210us) service_pool.cc:225] Handling call
0831 07:59:45.071499 (+ 57249us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:45.755112 (+683613us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:47.282251 (+1527139us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x7b540062f054. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:47.282416 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1570309888}
I20250831 07:59:47.188724 2574 raft_consensus.cc:738] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:47.237385 2648 raft_consensus.cc:2802] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 07:59:47.238896 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:47.214766 2516 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:47.239497 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 2016) took 1177 ms. Trace:
I20250831 07:59:47.239552 2448 raft_consensus.cc:2802] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:47.240316 2633 leader_election.cc:290] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:47.241684 2632 leader_election.cc:290] T 5eac48bce84e483582f863716d5c26f9 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:47.243006 2599 leader_election.cc:290] T be7a7e4ea6e1431ab37fb092f87cebd3 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:47.243528 2630 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:47.215754 2096 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:47.215461 2650 raft_consensus.cc:491] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:47.192080 2624 consensus_queue.cc:1035] T 3eb1971f5bb14c27bdbad34649f50ab4 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20250831 07:59:47.233188 2625 raft_consensus.cc:2802] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 07:59:47.244580 2593 leader_election.cc:290] T c3bdbd8499374432a94ff0909a4c5f95 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:47.244421 2532 raft_consensus.cc:513] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:47.241720 32464 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 5.
I20250831 07:59:47.245289 2513 raft_consensus.cc:513] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:47.245872 2563 leader_election.cc:290] T fc393e9ddc4c42dfa208cc7251b0b15b P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:47.225239 2651 raft_consensus.cc:491] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:47.226104 2610 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:47.277098 2530 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20250831 07:59:47.287681 2575 raft_consensus.cc:3058] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Advancing to term 10
I20250831 07:59:47.288043 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:47.200141 2623 raft_consensus.cc:491] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:47.187806 2595 raft_consensus.cc:3058] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65 [term 4 FOLLOWER]: Advancing to term 5
I20250831 07:59:46.973205 2514 consensus_queue.cc:237] T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:47.214947 2241 raft_consensus.cc:513] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:47.203088 2640 raft_consensus.cc:513] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:47.381187 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:47.585839 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 365233us
W20250831 07:59:47.458465 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 1.45103e+06us
I20250831 07:59:47.384433 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1562) took 1276 ms. Trace:
W20250831 07:59:47.469444 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:47.728389 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 465392us
W20250831 07:59:47.309813 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 61596.7us
I20250831 07:59:47.344327 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d3848dba46b4b239a92dc14f93ef48d" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:47.730569 31732 raft_consensus.cc:2466] T db51a15157664ec0a9d8402ca441f042 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:47.746817 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:47.833796 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:47.465093 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1642) took 1190 ms. Trace:
W20250831 07:59:47.933458 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:47.799010 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:48.126248 31774 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:47.727201 31848 leader_election.cc:336] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.654s (SENT)
I20250831 07:59:48.211231 31848 leader_election.cc:304] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, 7ecb257174554a26b2c35ad04b0eabae
W20250831 07:59:48.212390 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 485200us
I20250831 07:59:48.206516 31807 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 7.
I20250831 07:59:48.213593 31807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1803) took 1114 ms. Trace:
I20250831 07:59:48.214259 31807 rpcz_store.cc:276] 0831 07:59:47.099274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:47.833465 (+734191us) service_pool.cc:225] Handling call
0831 07:59:48.213519 (+380054us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:47.798911 31773 leader_election.cc:304] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7512ba2ed3a94982a22a75f51a647e65; no voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:48.217128 31773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 418286us
I20250831 07:59:48.206422 31731 rpcz_store.cc:276] 0831 07:59:46.274271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:46.399211 (+124940us) service_pool.cc:225] Handling call
0831 07:59:46.638996 (+239785us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:46.639169 (+ 173us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:46.639199 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:46.639240 (+ 41us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:46.639269 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:46.639317 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:47.464869 (+825552us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b540038d1d0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:47.465019 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":550678912}
I20250831 07:59:48.225167 31727 raft_consensus.cc:2408] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20250831 07:59:48.257898 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 524941us
I20250831 07:59:48.258708 31848 leader_election.cc:304] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:48.260612 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
W20250831 07:59:47.585777 2603 log.cc:927] Time spent T 2068a5cd20ba438a9dae813ac5acda03 P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 1.924s user 0.005s sys 0.005s
W20250831 07:59:47.931878 2652 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.003s sys 0.007s
W20250831 07:59:48.263999 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:48.267457 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:48.070924 2635 consensus_meta.cc:287] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.789s user 0.008s sys 0.004s
W20250831 07:59:48.271701 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:48.292886 31697 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:48.281955 31733 raft_consensus.cc:2466] T 95d9a42ca12741e382b05c0cd9007bfc P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 4.
I20250831 07:59:48.293526 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57688 (request call id 1652) took 1012 ms. Trace:
I20250831 07:59:47.602638 2561 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:47.729933 2648 raft_consensus.cc:491] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:48.042043 2513 leader_election.cc:290] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:47.933779 2457 leader_election.cc:290] T 2acd7b3ac5f94f53ac1ef93264415686 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:48.044343 2516 leader_election.cc:290] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:48.275736 2099 raft_consensus.cc:491] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:47.798614 31882 rpcz_store.cc:276] 0831 07:59:46.107918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:46.859394 (+751476us) service_pool.cc:225] Handling call
0831 07:59:46.873590 (+ 14196us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:46.878388 (+ 4798us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:46.878433 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:46.891617 (+ 13184us) log.cc:844] Serialized 26 byte log entry
0831 07:59:47.182117 (+290500us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:47.383657 (+201540us) log.cc:844] Serialized 14 byte log entry
0831 07:59:47.384008 (+ 351us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:47.384109 (+ 101us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:47.384168 (+ 59us) raft_consensus.cc:1683] finished
0831 07:59:47.384187 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:47.384357 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14103,"prepare.queue_time_us":981841,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":59,"raft.queue_time_us":30376,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":45770,"thread_start_us":409006,"threads_started":2,"wal-append.queue_time_us":407227}
I20250831 07:59:48.288678 32471 raft_consensus.cc:1273] T 90effb264f51485dac220ee32f49e4ad P 7ecb257174554a26b2c35ad04b0eabae [term 10 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20250831 07:59:48.276065 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:48.054961 2574 consensus_queue.cc:260] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:48.043140 2241 leader_election.cc:290] T 3054807582f54efba5d989f569ab0666 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:48.289249 32468 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcbe3794f3fc40819ffbbfef02a071c3" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:48.320446 32468 raft_consensus.cc:2408] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20250831 07:59:48.278417 31731 raft_consensus.cc:1238] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:48.290541 2640 leader_election.cc:290] T 660e7b2f56414f4fbd04a1929c99b390 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:47.450287 2582 raft_consensus.cc:491] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:47.796998 2633 raft_consensus.cc:491] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:47.744594 2623 raft_consensus.cc:3058] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:47.579530 2286 raft_consensus.cc:491] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:47.299875 2474 raft_consensus.cc:491] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:48.269187 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 2034) took 1048 ms. Trace:
I20250831 07:59:47.464077 2609 leader_election.cc:290] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:48.287218 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1565) took 2178 ms. Trace:
I20250831 07:59:47.875198 2557 leader_election.cc:290] T fe2bab0189fe475cb540bdf7a041d8b8 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:47.884424 2543 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:48.285415 32467 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:47.733356 2586 leader_election.cc:290] T b606cb7b5ccc4e2588ca98715c021620 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:48.282461 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:48.293931 31733 rpcz_store.cc:276] 0831 07:59:47.280816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:47.998649 (+717833us) service_pool.cc:225] Handling call
0831 07:59:48.293458 (+294809us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:48.292735 31729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 2006) took 2204 ms (client timeout 1929 ms). Trace:
I20250831 07:59:47.738536 2636 raft_consensus.cc:491] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:47.755599 2554 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:48.295521 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ea3cc82a6af45f0a86e4cc5c5a90335" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:48.234488 2508 raft_consensus.cc:2747] T 719b7e1bd5624bb9a08892e34c5e1b47 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 07:59:48.397554 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57678 (request call id 2030) took 1455 ms. Trace:
I20250831 07:59:48.401515 2452 raft_consensus.cc:3058] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:48.376201 2582 raft_consensus.cc:513] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:48.234484 2624 raft_consensus.cc:2747] T 6b2e336f43934f118ecacdc1e1d2da60 P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 07:59:48.403483 2582 leader_election.cc:290] T 9949d6ea1bea439fa2ca1fff1c5641bd P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:48.291636 2610 leader_election.cc:290] T 9560a731dd294178911faa26dbfc2ef8 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:47.744305 2651 raft_consensus.cc:3058] T b606cb7b5ccc4e2588ca98715c021620 P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:47.738363 2618 leader_election.cc:290] T 9ea3cc82a6af45f0a86e4cc5c5a90335 P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027)
I20250831 07:59:47.729827 2448 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:48.415190 2636 raft_consensus.cc:513] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:48.416065 2448 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Advancing to term 8
I20250831 07:59:48.257230 2625 raft_consensus.cc:491] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:48.261117 2275 raft_consensus.cc:491] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:48.483495 31807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1975) took 1168 ms (client timeout 1552 ms). Trace:
W20250831 07:59:48.484031 31807 rpcz_store.cc:269] 0831 07:59:47.314870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.264457 (+949587us) service_pool.cc:225] Handling call
0831 07:59:48.483424 (+218967us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:48.298188 2593 raft_consensus.cc:491] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:48.308718 2660 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:48.301496 2563 raft_consensus.cc:491] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:48.311753 2096 leader_election.cc:290] T 2b32e93c114645eeb4e707778c91e12e P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:48.305514 2648 raft_consensus.cc:3058] T 409f8ff1108b412382695bc6ce64e3cc P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Advancing to term 7
W20250831 07:59:48.494252 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:48.502562 31698 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.217.4:43319 timed out after 1.868s (SENT) [suppressed 26 similar messages]
I20250831 07:59:48.312059 31804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39084 (request call id 1777) took 4273 ms. Trace:
I20250831 07:59:48.287660 2632 raft_consensus.cc:491] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:48.280791 2652 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.893s user 0.004s sys 0.007s
W20250831 07:59:48.271687 2525 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.000s sys 0.013s
I20250831 07:59:48.057153 2532 leader_election.cc:290] T 91390ee2f1e2445ca47e8cb7bd8eeecd P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 election: Requested vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
W20250831 07:59:48.317514 31875 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.002s user 0.014s sys 0.010s
I20250831 07:59:48.312780 2569 raft_consensus.cc:491] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:48.266309 2506 consensus_queue.cc:1035] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20250831 07:59:48.486375 2557 raft_consensus.cc:491] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:48.319161 31849 leader_election.cc:304] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
W20250831 07:59:48.513664 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 194574us
I20250831 07:59:47.496372 2392 consensus_queue.cc:260] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:48.329212 31731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1644) took 1081 ms. Trace:
I20250831 07:59:48.516364 31731 rpcz_store.cc:276] 0831 07:59:47.247330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.271234 (+1023904us) service_pool.cc:225] Handling call
0831 07:59:48.271522 (+ 288us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:48.328920 (+ 57398us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.329147 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:48.517752 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "000f9ab653554f498fe14065e6656a62" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:48.518553 31731 raft_consensus.cc:2466] T 000f9ab653554f498fe14065e6656a62 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:48.312713 2513 raft_consensus.cc:491] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:48.367638 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7c92dde8fc9490dafb6d51ad8b26855" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:48.262576 2650 raft_consensus.cc:513] T bb2667d2ec934433ac10d395c1d29db4 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:48.306602 2561 raft_consensus.cc:513] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:48.401795 31730 rpcz_store.cc:276] 0831 07:59:46.941904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:46.942022 (+ 118us) service_pool.cc:225] Handling call
0831 07:59:48.397209 (+1455187us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b5400091854. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:48.397471 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30695552}
I20250831 07:59:47.840021 32465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38366 (request call id 565) took 1169 ms. Trace:
I20250831 07:59:48.318351 2099 raft_consensus.cc:513] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:48.404016 31879 rpcz_store.cc:276] 0831 07:59:46.108820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:47.201391 (+1092571us) service_pool.cc:225] Handling call
0831 07:59:47.201622 (+ 231us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:47.201956 (+ 334us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:47.201984 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:47.217644 (+ 15660us) log.cc:844] Serialized 26 byte log entry
0831 07:59:47.580685 (+363041us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:47.580743 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:47.580837 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:48.287012 (+706175us) raft_consensus.cc:1683] finished
0831 07:59:48.287035 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:48.287162 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2035,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":116,"thread_start_us":338379,"threads_started":1,"wal-append.queue_time_us":871694}
I20250831 07:59:48.535519 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:48.536168 32466 raft_consensus.cc:3058] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 3 FOLLOWER]: Advancing to term 4
I20250831 07:59:48.536193 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "064c41a8e5dc42e8bdb35225c6aae643" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:48.537334 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1579) took 1304 ms. Trace:
I20250831 07:59:48.537796 31879 rpcz_store.cc:276] 0831 07:59:47.232613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.535876 (+1303263us) service_pool.cc:225] Handling call
0831 07:59:48.537256 (+ 1380us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:48.538590 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 237775us
I20250831 07:59:48.539212 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2f76351ade3435092b6fac201772424" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:48.387135 32463 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38362 (request call id 448) took 1829 ms. Trace:
I20250831 07:59:48.302274 2635 raft_consensus.cc:513] T 4d055e59fc97471a93e491ee81afa361 P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:48.404067 31729 rpcz_store.cc:269] 0831 07:59:46.059898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:46.130423 (+ 70525us) service_pool.cc:225] Handling call
0831 07:59:48.264570 (+2134147us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b540038ccd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:48.264686 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":102081280}
I20250831 07:59:48.316076 2516 raft_consensus.cc:2747] T 9d3848dba46b4b239a92dc14f93ef48d P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250831 07:59:48.387689 31800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43428 (request call id 498) took 1824 ms (client timeout 2036 ms). Trace:
I20250831 07:59:48.405864 31727 rpcz_store.cc:276] 0831 07:59:47.220497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.268270 (+1047773us) service_pool.cc:225] Handling call
0831 07:59:48.268639 (+ 369us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:48.268828 (+ 189us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:48.268852 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:48.268897 (+ 45us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:48.268927 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.268977 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:48.269116 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:48.541733 2392 raft_consensus.cc:3058] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Advancing to term 7
I20250831 07:59:48.549695 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1649) took 1269 ms. Trace:
I20250831 07:59:48.550093 31727 rpcz_store.cc:276] 0831 07:59:47.280014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.548928 (+1268914us) service_pool.cc:225] Handling call
0831 07:59:48.549226 (+ 298us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:48.549405 (+ 179us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:48.549428 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:48.549469 (+ 41us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:48.549496 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.549549 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:48.549639 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:48.485828 31733 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:48.552325 31733 raft_consensus.cc:2466] T 5eac48bce84e483582f863716d5c26f9 P a0d845d01c8c4894895f66e3beae298c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:48.504751 2567 consensus_queue.cc:1046] T 9a40df3e72bd462091ca307d230a90ed P a0d845d01c8c4894895f66e3beae298c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20250831 07:59:48.398686 2587 raft_consensus.cc:3053] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Stepping down as leader of term 5
I20250831 07:59:48.558557 31727 raft_consensus.cc:1238] T ccd51d2e599e4535ad0f2f6afcb1bfee P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 6. Current term is 7. Ops: []
I20250831 07:59:48.504290 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1626) took 3669 ms. Trace:
I20250831 07:59:48.503453 31804 rpcz_store.cc:276] 0831 07:59:44.038407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.038532 (+ 125us) service_pool.cc:225] Handling call
0831 07:59:44.605691 (+567159us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b5800245198. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce505e251 00007fdce505e5ce 000000000035a3c9 00007fdcecee5300 00007fdcecfabec4 00007fdcecfabf31 00007fdcecf6ef50 00007fdcecf5a199 00007fdcecf59c42 00007fdce65e6b44
0831 07:59:46.658391 (+2052700us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:47.176489 (+518098us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.309628 (+1133139us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b5400235d54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:48.311771 (+ 2143us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b5400235d50. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:48.311964 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5520819072}
I20250831 07:59:48.386508 2475 raft_consensus.cc:491] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:48.404114 2286 raft_consensus.cc:513] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:48.319715 2647 consensus_queue.cc:1046] T 4f319de3ac3c4b9a9b87cd497e02c51c P a0d845d01c8c4894895f66e3beae298c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.175s
I20250831 07:59:48.485414 2625 raft_consensus.cc:3058] T e93b827594384bcf80c7487b061b4c24 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Advancing to term 8
W20250831 07:59:48.510332 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:48.565536 31804 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "953dd19ad1e4414ab1c956437fd8bbfa" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:48.511399 31847 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.010s sys 0.048s
W20250831 07:59:48.566135 31847 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.761s user 0.010s sys 0.049s
W20250831 07:59:48.566978 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 768784us
W20250831 07:59:48.506063 2657 log.cc:927] Time spent T 409f8ff1108b412382695bc6ce64e3cc P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20250831 07:59:48.285166 32462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eac48bce84e483582f863716d5c26f9" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:48.512676 31848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 254043us
I20250831 07:59:48.542938 2554 raft_consensus.cc:491] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:48.521176 31878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1923) took 2798 ms (client timeout 1693 ms). Trace:
I20250831 07:59:48.521746 31876 raft_consensus.cc:1238] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Rejecting Update request from peer 7512ba2ed3a94982a22a75f51a647e65 for earlier term 4. Current term is 5. Ops: []
I20250831 07:59:48.486840 2609 raft_consensus.cc:491] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:48.522989 31882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1947) took 1305 ms (client timeout 1693 ms). Trace:
W20250831 07:59:48.590862 31882 rpcz_store.cc:269] 0831 07:59:47.217452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.317285 (+1099833us) service_pool.cc:225] Handling call
0831 07:59:48.522915 (+205630us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:48.592216 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db51a15157664ec0a9d8402ca441f042" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:48.593232 31882 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1576) took 1365 ms. Trace:
I20250831 07:59:48.593690 31882 rpcz_store.cc:276] 0831 07:59:47.227892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.591855 (+1363963us) service_pool.cc:225] Handling call
0831 07:59:48.593154 (+ 1299us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:48.405062 2633 raft_consensus.cc:513] T e3a3b559f26e43f8942b8753f3e30607 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:48.510121 2525 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.894s user 0.000s sys 0.013s
I20250831 07:59:48.532169 2557 raft_consensus.cc:513] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:48.533324 31699 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:48.539979 32463 rpcz_store.cc:276] 0831 07:59:46.558121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:46.558310 (+ 189us) service_pool.cc:225] Handling call
0831 07:59:46.647238 (+ 88928us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:46.647502 (+ 264us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:46.647541 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:46.647613 (+ 72us) raft_consensus.cc:1643] Marking committed up to 8
0831 07:59:46.647666 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:46.647745 (+ 79us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:48.386946 (+1739201us) spinlock_profiling.cc:243] Waited 80.7 ms on lock 0x7b54005ad2d0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:48.387077 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":185561472}
I20250831 07:59:48.485634 2474 raft_consensus.cc:513] T 54f9ace15ae4422682ce241dda87dedf P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:48.541651 32471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:48.542311 31879 raft_consensus.cc:2408] T f2f76351ade3435092b6fac201772424 P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7512ba2ed3a94982a22a75f51a647e65 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250831 07:59:48.503530 2563 raft_consensus.cc:513] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:48.542492 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1645) took 1294 ms. Trace:
I20250831 07:59:48.616353 31730 rpcz_store.cc:276] 0831 07:59:47.247612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.541644 (+1294032us) service_pool.cc:225] Handling call
0831 07:59:48.541957 (+ 313us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:48.542145 (+ 188us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:48.542175 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:48.542221 (+ 46us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:48.542258 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.542324 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:48.542436 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:48.554879 31733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1651) took 1274 ms. Trace:
I20250831 07:59:48.617482 31733 rpcz_store.cc:276] 0831 07:59:47.280584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.553124 (+1272540us) service_pool.cc:225] Handling call
0831 07:59:48.553890 (+ 766us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:48.554212 (+ 322us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:48.554253 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:48.554320 (+ 67us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:48.554383 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.554637 (+ 254us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:48.554803 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:48.488777 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "719b7e1bd5624bb9a08892e34c5e1b47" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:48.619093 31807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1957) took 1386 ms (client timeout 1654 ms). Trace:
W20250831 07:59:48.619557 31807 rpcz_store.cc:269] 0831 07:59:47.232972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.488415 (+1255443us) service_pool.cc:225] Handling call
0831 07:59:48.619024 (+130609us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:48.547756 31800 rpcz_store.cc:269] 0831 07:59:46.562798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:46.563788 (+ 990us) service_pool.cc:225] Handling call
0831 07:59:48.387510 (+1823722us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b540023ffd4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:48.387639 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1513432320}
I20250831 07:59:48.500545 2593 raft_consensus.cc:513] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:48.370563 2514 raft_consensus.cc:491] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8247a38b550649aa9cd6e7668dcaec10)
I20250831 07:59:48.500193 2660 raft_consensus.cc:513] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:48.559424 31726 rpcz_store.cc:276] 0831 07:59:44.835291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.835446 (+ 155us) service_pool.cc:225] Handling call
0831 07:59:44.890661 (+ 55215us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:44.976935 (+ 86274us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:44.976988 (+ 53us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:45.122413 (+145425us) log.cc:844] Serialized 26 byte log entry
0831 07:59:45.256951 (+134538us) raft_consensus.cc:1643] Marking committed up to 4
0831 07:59:45.257039 (+ 88us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:45.257241 (+ 202us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:47.421210 (+2163969us) spinlock_profiling.cc:243] Waited 82.4 ms on lock 0x7b440002304c. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce4edec41 00007fdce4fad2b5 00007fdce4fadcd2 00007fdce7c4d231 00007fdce7c48b5a 00007fdce7c3c91e 00007fdce7c3bc66 00007fdce7c3b888 00007fdce7c452c1 00007fdce7c45279 00007fdce7c45209
0831 07:59:48.488541 (+1067331us) raft_consensus.cc:1683] finished
0831 07:59:48.488572 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:48.504223 (+ 15651us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":231298,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"spinlock_wait_cycles":189466368,"thread_start_us":219485,"threads_started":2,"wal-append.queue_time_us":391932}
I20250831 07:59:48.561929 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57688 (request call id 1650) took 1281 ms. Trace:
I20250831 07:59:48.531494 2582 raft_consensus.cc:491] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
W20250831 07:59:48.657706 31874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51246 (request call id 336) took 3838 ms (client timeout 1671 ms). Trace:
I20250831 07:59:48.531250 2668 raft_consensus.cc:2802] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 07:59:48.565696 32435 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:48.511750 31875 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.196s user 0.015s sys 0.010s
I20250831 07:59:48.511162 2632 raft_consensus.cc:513] T 2b32e93c114645eeb4e707778c91e12e P 7512ba2ed3a94982a22a75f51a647e65 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:48.573376 31847 leader_election.cc:304] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae, 8247a38b550649aa9cd6e7668dcaec10; no voters: a0d845d01c8c4894895f66e3beae298c
W20250831 07:59:48.573648 31848 leader_election.cc:336] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261): Timed out: RequestConsensusVote RPC to 127.29.217.2:37261 timed out after 1.755s (SENT)
I20250831 07:59:48.672322 31848 leader_election.cc:304] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10; no voters: 7512ba2ed3a94982a22a75f51a647e65, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:48.519562 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3d650c422ee41af95705f3fd876ddb8" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:48.489370 2275 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:48.579538 31878 rpcz_store.cc:269] 0831 07:59:45.722392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:45.723393 (+ 1001us) service_pool.cc:225] Handling call
0831 07:59:48.521002 (+2797609us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b5400058054. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:48.521112 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":293837056}
I20250831 07:59:48.674014 31878 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:48.674629 31878 raft_consensus.cc:2466] T 982febd23d7540aaa3b48b70133e982e P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d845d01c8c4894895f66e3beae298c in term 8.
I20250831 07:59:48.675237 31878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57346 (request call id 1949) took 1456 ms. Trace:
I20250831 07:59:48.521474 2569 raft_consensus.cc:513] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:48.612236 2335 raft_consensus.cc:491] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
W20250831 07:59:48.608389 31698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 105875us
I20250831 07:59:48.415266 2543 raft_consensus.cc:513] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:48.533380 2513 raft_consensus.cc:513] T 9d3848dba46b4b239a92dc14f93ef48d P a0d845d01c8c4894895f66e3beae298c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:48.564468 2595 consensus_meta.cc:287] T 26525016efb247c3b1c2bf7fec0e1658 P 7512ba2ed3a94982a22a75f51a647e65: Time spent flushing consensus metadata: real 0.820s user 0.012s sys 0.000s
I20250831 07:59:48.636083 2593 leader_election.cc:290] T f611ea44f17c403e818574844e13b946 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:48.640895 2557 leader_election.cc:290] T 9e12d332258d4d0b8d0421b49ed5d0c0 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261)
I20250831 07:59:48.614676 32471 raft_consensus.cc:2391] T 4f319de3ac3c4b9a9b87cd497e02c51c P 7ecb257174554a26b2c35ad04b0eabae [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 10: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
I20250831 07:59:48.558991 2508 consensus_queue.cc:1035] T db51a15157664ec0a9d8402ca441f042 P 8247a38b550649aa9cd6e7668dcaec10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20250831 07:59:48.623368 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1580) took 1386 ms. Trace:
I20250831 07:59:48.706777 31879 rpcz_store.cc:276] 0831 07:59:47.236909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.538914 (+1302005us) service_pool.cc:225] Handling call
0831 07:59:48.623297 (+ 84383us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:48.649955 31727 rpcz_store.cc:276] 0831 07:59:47.280317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.551333 (+1271016us) service_pool.cc:225] Handling call
0831 07:59:48.551707 (+ 374us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:48.561481 (+ 9774us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.561849 (+ 368us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:48.637607 31882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95d9a42ca12741e382b05c0cd9007bfc" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:48.639379 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a62a36682136474ca552fbc53190e09e" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:48.663741 2668 raft_consensus.cc:491] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 07:59:48.631632 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "409f8ff1108b412382695bc6ce64e3cc" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:48.640347 2517 consensus_queue.cc:1046] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
W20250831 07:59:48.729202 32464 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38366 (request call id 564) took 2232 ms (client timeout 1868 ms). Trace:
I20250831 07:59:48.535404 32465 rpcz_store.cc:276] 0831 07:59:46.627890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:46.639146 (+ 11256us) service_pool.cc:225] Handling call
0831 07:59:46.639486 (+ 340us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:46.902157 (+262671us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:46.907786 (+ 5629us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:46.918781 (+ 10995us) raft_consensus.cc:1643] Marking committed up to 6
0831 07:59:46.923681 (+ 4900us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:47.264041 (+340360us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b5400541ad4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:47.264089 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:47.738675 (+474586us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":576636160}
I20250831 07:59:48.668820 31880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1922) took 2939 ms. Trace:
I20250831 07:59:48.735699 31880 rpcz_store.cc:276] 0831 07:59:45.722070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:45.725730 (+ 3660us) service_pool.cc:225] Handling call
0831 07:59:45.751700 (+ 25970us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:46.063717 (+312017us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.661263 (+2597546us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b5400245754. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:48.661495 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":517422336}
I20250831 07:59:48.673154 31731 raft_consensus.cc:2466] T d3d650c422ee41af95705f3fd876ddb8 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 7.
W20250831 07:59:48.632320 2666 log.cc:927] Time spent T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 0.126s user 0.000s sys 0.001s
I20250831 07:59:48.675568 31878 rpcz_store.cc:276] 0831 07:59:47.219197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.673759 (+1454562us) service_pool.cc:225] Handling call
0831 07:59:48.675168 (+ 1409us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:48.632896 2671 log.cc:927] Time spent T 54f9ace15ae4422682ce241dda87dedf P a0d845d01c8c4894895f66e3beae298c: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
W20250831 07:59:48.641184 2674 log.cc:927] Time spent T 9a40df3e72bd462091ca307d230a90ed P 7ecb257174554a26b2c35ad04b0eabae: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20250831 07:59:48.633396 2673 log.cc:927] Time spent T c3bdbd8499374432a94ff0909a4c5f95 P 7512ba2ed3a94982a22a75f51a647e65: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
W20250831 07:59:48.677975 31700 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 07:59:48.637497 2644 log.cc:927] Time spent T 660e7b2f56414f4fbd04a1929c99b390 P 8247a38b550649aa9cd6e7668dcaec10: Append to log took a long time: real 1.695s user 0.005s sys 0.012s
W20250831 07:59:48.680157 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:48.711059 2241 raft_consensus.cc:491] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7512ba2ed3a94982a22a75f51a647e65)
I20250831 07:59:48.565598 2475 raft_consensus.cc:513] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
W20250831 07:59:48.607669 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 69565.3us
I20250831 07:59:48.605804 2554 raft_consensus.cc:513] T 855219eabb9f4417ad63ec7efa7982da P 8247a38b550649aa9cd6e7668dcaec10 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:48.609166 2609 raft_consensus.cc:513] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:48.766362 31877 raft_consensus.cc:1273] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Refusing update from remote peer 7512ba2ed3a94982a22a75f51a647e65: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250831 07:59:48.766461 2609 leader_election.cc:290] T 3f70249b7dd14c0e804b05303fa06f09 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:48.766790 31877 pending_rounds.cc:85] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10: Aborting all ops after (but not including) 3
I20250831 07:59:48.642716 2561 leader_election.cc:290] T 9d510d241da34d25a50f2d65d17041a0 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d845d01c8c4894895f66e3beae298c (127.29.217.1:45027), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:48.704309 2686 raft_consensus.cc:491] T 90effb264f51485dac220ee32f49e4ad P a0d845d01c8c4894895f66e3beae298c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:48.767076 31877 pending_rounds.cc:107] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250831 07:59:48.705054 2275 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 7512ba2ed3a94982a22a75f51a647e65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:48.709817 31800 raft_consensus.cc:2408] T a62a36682136474ca552fbc53190e09e P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250831 07:59:48.767386 2629 raft_consensus.cc:3053] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Stepping down as leader of term 6
W20250831 07:59:48.767791 31800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1800) took 1678 ms (client timeout 1840 ms). Trace:
I20250831 07:59:48.767848 2629 raft_consensus.cc:738] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7512ba2ed3a94982a22a75f51a647e65, State: Running, Role: LEADER
I20250831 07:59:48.639719 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26525016efb247c3b1c2bf7fec0e1658" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:48.768460 32466 raft_consensus.cc:2466] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:48.769035 32463 raft_consensus.cc:2408] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250831 07:59:48.694972 2335 raft_consensus.cc:513] T 518e6f5a91554de78e0bd0b6ec04c12a P 7512ba2ed3a94982a22a75f51a647e65 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
W20250831 07:59:48.710551 31847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.4:43319, user_credentials={real_user=slave}} blocked reactor thread for 137262us
I20250831 07:59:48.729959 2668 raft_consensus.cc:3058] T 95d9a42ca12741e382b05c0cd9007bfc P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:48.707746 31879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a98f72c022fe465da362553182e4ecca" candidate_uuid: "7512ba2ed3a94982a22a75f51a647e65" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10"
I20250831 07:59:48.777433 31879 raft_consensus.cc:3058] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Advancing to term 6
I20250831 07:59:48.643303 2525 consensus_queue.cc:1046] T f7c92dde8fc9490dafb6d51ad8b26855 P 7512ba2ed3a94982a22a75f51a647e65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
W20250831 07:59:48.716562 31807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1812) took 1476 ms (client timeout 1682 ms). Trace:
W20250831 07:59:48.780068 31807 rpcz_store.cc:269] 0831 07:59:47.240190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.631321 (+1391131us) service_pool.cc:225] Handling call
0831 07:59:48.716481 (+ 85160us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:48.658155 31874 rpcz_store.cc:269] 0831 07:59:44.819222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:44.819507 (+ 285us) service_pool.cc:225] Handling call
0831 07:59:48.657526 (+3838019us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b54003efb54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:48.657638 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2133433600}
I20250831 07:59:48.716828 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:48.729583 31801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43428 (request call id 493) took 2244 ms. Trace:
I20250831 07:59:48.783699 31726 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P a0d845d01c8c4894895f66e3beae298c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 4.
I20250831 07:59:48.783612 31801 rpcz_store.cc:276] 0831 07:59:46.485336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:46.485540 (+ 204us) service_pool.cc:225] Handling call
0831 07:59:46.485839 (+ 299us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:46.844919 (+359080us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:46.844950 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:46.844992 (+ 42us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:46.845024 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.729349 (+1884325us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b5400028854. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:48.729398 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:48.729531 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":825400192}
I20250831 07:59:48.666306 31875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57340 (request call id 1564) took 2557 ms. Trace:
I20250831 07:59:48.784840 31875 rpcz_store.cc:276] 0831 07:59:46.108544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:46.881204 (+772660us) service_pool.cc:225] Handling call
0831 07:59:46.881584 (+ 380us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:47.075497 (+193913us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:47.075547 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:47.113053 (+ 37506us) log.cc:844] Serialized 26 byte log entry
0831 07:59:47.230155 (+117102us) raft_consensus.cc:1643] Marking committed up to 5
0831 07:59:48.665809 (+1435654us) log.cc:844] Serialized 14 byte log entry
0831 07:59:48.665997 (+ 188us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.666072 (+ 75us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:48.666108 (+ 36us) raft_consensus.cc:1683] finished
0831 07:59:48.666120 (+ 12us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:48.666254 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":958207,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":62039,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":27019,"thread_start_us":1596552,"threads_started":3,"wal-append.queue_time_us":1152083}
I20250831 07:59:48.686007 2652 raft_consensus.cc:491] T f7c92dde8fc9490dafb6d51ad8b26855 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:48.729687 32464 rpcz_store.cc:269] 0831 07:59:46.477889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:46.478086 (+ 197us) service_pool.cc:225] Handling call
0831 07:59:48.710476 (+2232390us) spinlock_profiling.cc:243] Waited 645 ms on lock 0x7b5400144e54. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce80012aa 00007fdcecf5a5c7 00007fdce65e90c4 00007fdce65e9052 00007fdce65e8f81 00007fdce65e8efc 00007fdce65e81b2 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:48.710593 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1484465024}
I20250831 07:59:48.709506 2661 raft_consensus.cc:491] T 9d510d241da34d25a50f2d65d17041a0 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7ecb257174554a26b2c35ad04b0eabae)
I20250831 07:59:48.633818 2675 consensus_queue.cc:1035] T bb2667d2ec934433ac10d395c1d29db4 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20250831 07:59:48.741073 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3a3b559f26e43f8942b8753f3e30607" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:48.672394 2582 raft_consensus.cc:513] T 14ab15c60aaf42cf911851425eaab94d P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:48.741694 2660 leader_election.cc:290] T 9e3be3f093d144268837fd81ceb8b44a P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:48.741961 31880 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8247a38b550649aa9cd6e7668dcaec10" is_pre_election: true
I20250831 07:59:48.742489 2563 leader_election.cc:290] T 1e9fcfb2ddfa4c6eb4e9264f55b5dba0 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:48.660324 2514 raft_consensus.cc:513] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } } peers { permanent_uuid: "7ecb257174554a26b2c35ad04b0eabae" member_type: VOTER last_known_addr { host: "127.29.217.4" port: 43319 } attrs { promote: false } }
I20250831 07:59:48.705660 2630 raft_consensus.cc:491] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 07:59:48.752734 31775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 139403us
I20250831 07:59:48.815423 31731 raft_consensus.cc:2391] T e3a3b559f26e43f8942b8753f3e30607 P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 7: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:48.835093 2630 raft_consensus.cc:513] T 9560a731dd294178911faa26dbfc2ef8 P 7512ba2ed3a94982a22a75f51a647e65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:48.785585 2583 consensus_queue.cc:1046] T 9949d6ea1bea439fa2ca1fff1c5641bd P 7ecb257174554a26b2c35ad04b0eabae [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20250831 07:59:48.682734 31881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57346 (request call id 1925) took 2761 ms. Trace:
I20250831 07:59:48.730628 2516 raft_consensus.cc:2802] T fe2bab0189fe475cb540bdf7a041d8b8 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Leader election won for term 7
W20250831 07:59:48.758069 32435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.2:37261, user_credentials={real_user=slave}} blocked reactor thread for 92857.9us
I20250831 07:59:48.761577 32436 leader_election.cc:304] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:48.762650 2677 raft_consensus.cc:2747] T be7a7e4ea6e1431ab37fb092f87cebd3 P 8247a38b550649aa9cd6e7668dcaec10 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 07:59:48.558892 2587 raft_consensus.cc:738] T 3eb1971f5bb14c27bdbad34649f50ab4 P a0d845d01c8c4894895f66e3beae298c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d845d01c8c4894895f66e3beae298c, State: Running, Role: LEADER
I20250831 07:59:48.767362 31877 raft_consensus.cc:2887] T 3f70249b7dd14c0e804b05303fa06f09 P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250831 07:59:48.772724 31847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:48.708580 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
W20250831 07:59:48.768277 31800 rpcz_store.cc:269] 0831 07:59:47.089398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.620799 (+1531401us) service_pool.cc:225] Handling call
0831 07:59:48.767732 (+146933us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:48.841130 32436 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 79636.6us
I20250831 07:59:48.806735 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:48.807293 31804 raft_consensus.cc:2408] T 953dd19ad1e4414ab1c956437fd8bbfa P 7512ba2ed3a94982a22a75f51a647e65 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 07:59:48.808185 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d77811606c643a4970bc6b8a5f39fdb" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65"
I20250831 07:59:48.843150 31801 raft_consensus.cc:2391] T 6d77811606c643a4970bc6b8a5f39fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d845d01c8c4894895f66e3beae298c in current term 9: Already voted for candidate 7ecb257174554a26b2c35ad04b0eabae in this term.
W20250831 07:59:48.843783 31801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39084 (request call id 1795) took 1767 ms (client timeout 1898 ms). Trace:
I20250831 07:59:48.840665 32435 leader_election.cc:304] T 26525016efb247c3b1c2bf7fec0e1658 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 7512ba2ed3a94982a22a75f51a647e65, 8247a38b550649aa9cd6e7668dcaec10
W20250831 07:59:48.844108 31801 rpcz_store.cc:269] 0831 07:59:47.076073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.807903 (+1731830us) service_pool.cc:225] Handling call
0831 07:59:48.843705 (+ 35802us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:48.806666 31698 leader_election.cc:304] T 982febd23d7540aaa3b48b70133e982e P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:48.768934 2543 leader_election.cc:290] T 59e33bfa463944f89cfbb00d1247323a P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:48.835747 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a187e6c5ed8f4214b4486aab25955909" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:48.837878 31881 rpcz_store.cc:276] 0831 07:59:45.921740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:45.931103 (+ 9363us) service_pool.cc:225] Handling call
0831 07:59:46.032600 (+101497us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:46.033214 (+ 614us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.682532 (+2649318us) spinlock_profiling.cc:243] Waited 85 ms on lock 0x7b540007d0d0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:48.682679 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":195425024}
I20250831 07:59:48.838198 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38366 (request call id 556) took 2917 ms. Trace:
I20250831 07:59:48.779659 2561 raft_consensus.cc:491] T 9e3be3f093d144268837fd81ceb8b44a P 8247a38b550649aa9cd6e7668dcaec10 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 07:59:48.839711 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe2bab0189fe475cb540bdf7a041d8b8" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:48.803423 31879 raft_consensus.cc:2466] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7512ba2ed3a94982a22a75f51a647e65 in term 6.
I20250831 07:59:48.856257 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 2031) took 1653 ms. Trace:
I20250831 07:59:48.857136 31875 raft_consensus.cc:1238] T a98f72c022fe465da362553182e4ecca P 8247a38b550649aa9cd6e7668dcaec10 [term 6 FOLLOWER]: Rejecting Update request from peer a0d845d01c8c4894895f66e3beae298c for earlier term 5. Current term is 6. Ops: []
W20250831 07:59:48.856828 31879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57340 (request call id 1578) took 1626 ms (client timeout 2085 ms). Trace:
W20250831 07:59:48.858318 31879 rpcz_store.cc:269] 0831 07:59:47.229946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.707457 (+1477511us) service_pool.cc:225] Handling call
0831 07:59:48.856754 (+149297us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:48.858690 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:48.859705 31731 raft_consensus.cc:2391] T 9949d6ea1bea439fa2ca1fff1c5641bd P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 9: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
W20250831 07:59:48.860474 31849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.217.1:45027, user_credentials={real_user=slave}} blocked reactor thread for 170625us
I20250831 07:59:48.845861 31807 raft_consensus.cc:2366] T 0cbf2ae2189542948a3ffb6b88a4fa21 P 7512ba2ed3a94982a22a75f51a647e65 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 for earlier term 5. Current term is 6.
I20250831 07:59:48.861630 31849 leader_election.cc:304] T ca7b8b0d72c94ed386b217b936955022 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c; no voters:
I20250831 07:59:48.856820 31728 rpcz_store.cc:276] 0831 07:59:47.203264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:47.219562 (+ 16298us) service_pool.cc:225] Handling call
0831 07:59:47.230635 (+ 11073us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 07:59:47.582130 (+351495us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:47.582165 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 07:59:47.603478 (+ 21313us) log.cc:844] Serialized 26 byte log entry
0831 07:59:48.313252 (+709774us) raft_consensus.cc:1643] Marking committed up to 3
0831 07:59:48.313306 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.313394 (+ 88us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 07:59:48.855977 (+542583us) raft_consensus.cc:1683] finished
0831 07:59:48.856009 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:48.856179 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16710,"raft.queue_time_us":35578,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":114,"thread_start_us":683653,"threads_started":2,"wal-append.queue_time_us":678817}
W20250831 07:59:48.862473 31807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1955) took 1770 ms (client timeout 1860 ms). Trace:
I20250831 07:59:48.847157 32469 rpcz_store.cc:276] 0831 07:59:45.921096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:45.921215 (+ 119us) service_pool.cc:225] Handling call
0831 07:59:46.021018 (+ 99803us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:46.240109 (+219091us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.815546 (+2575437us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54004c13d4. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffddc1 00007fdce7ffc7fe 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106
0831 07:59:48.838017 (+ 22471us) spinlock_profiling.cc:243] Waited 89.8 ms on lock 0x7b54004c13d0. stack: 00007fdce5069823 00007fdce5069720 00007fdce49c3574 00007fdcee4c5784 00007fdcee4c5719 00007fdce7f816e1 00007fdce7ffc86d 00007fdcecf59c89 00007fdce65e6b44 00007fdce65e6ad2 00007fdce65e6a01 00007fdce65e697c 00007fdce65e5c32 00007fdce63fcccc 00007fdce63fc106 00007fdce63fbaaf
0831 07:59:48.838142 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":636839168}
I20250831 07:59:48.849867 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7a7e4ea6e1431ab37fb092f87cebd3" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:48.862867 31807 rpcz_store.cc:269] 0831 07:59:47.091523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.806412 (+1714889us) service_pool.cc:225] Handling call
0831 07:59:48.862292 (+ 55880us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:48.863890 31801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39082 (request call id 1961) took 1617 ms (client timeout 1755 ms). Trace:
I20250831 07:59:48.854071 32466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cbf2ae2189542948a3ffb6b88a4fa21" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae"
I20250831 07:59:48.863929 31807 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "982febd23d7540aaa3b48b70133e982e" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
W20250831 07:59:48.871017 31801 rpcz_store.cc:269] 0831 07:59:47.246103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.849539 (+1603436us) service_pool.cc:225] Handling call
0831 07:59:48.863775 (+ 14236us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 07:59:48.852638 32434 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:48.872236 31801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d573ed55e048098c90bcbb00d12fdb" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:48.768849 2629 consensus_queue.cc:260] T 838ecfd85da44202b231fff2105cde0b P 7512ba2ed3a94982a22a75f51a647e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:48.868733 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 353) took 2276 ms. Trace:
I20250831 07:59:48.874969 32464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca7b8b0d72c94ed386b217b936955022" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
W20250831 07:59:48.869496 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:48.869628 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57678 (request call id 2037) took 1576 ms. Trace:
I20250831 07:59:48.875643 32464 raft_consensus.cc:2466] T ca7b8b0d72c94ed386b217b936955022 P 7ecb257174554a26b2c35ad04b0eabae [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
W20250831 07:59:48.876108 31698 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 07:59:48.875371 31879 rpcz_store.cc:276] 0831 07:59:46.592244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.867890 (+2275646us) service_pool.cc:225] Handling call
0831 07:59:48.868202 (+ 312us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:48.868402 (+ 200us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 07:59:48.868437 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:48.868489 (+ 52us) raft_consensus.cc:1643] Marking committed up to 8
0831 07:59:48.868521 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.868578 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:48.868678 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 07:59:48.769709 2241 raft_consensus.cc:513] T 26b32ddce4444eb89ebd7a73a7900bdf P a0d845d01c8c4894895f66e3beae298c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d845d01c8c4894895f66e3beae298c" member_type: VOTER last_known_addr { host: "127.29.217.1" port: 45027 } } peers { permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 } } peers { permanent_uuid: "8247a38b550649aa9cd6e7668dcaec10" member_type: VOTER last_known_addr { host: "127.29.217.3" port: 42585 } }
I20250831 07:59:48.860532 31731 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f319de3ac3c4b9a9b87cd497e02c51c" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c"
I20250831 07:59:48.873180 31801 raft_consensus.cc:2391] T 25d573ed55e048098c90bcbb00d12fdb P 7512ba2ed3a94982a22a75f51a647e65 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8247a38b550649aa9cd6e7668dcaec10 in current term 11: Already voted for candidate a0d845d01c8c4894895f66e3beae298c in this term.
I20250831 07:59:48.875313 32463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9949d6ea1bea439fa2ca1fff1c5641bd" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:48.876055 31730 rpcz_store.cc:276] 0831 07:59:47.293488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 07:59:48.627965 (+1334477us) service_pool.cc:225] Handling call
0831 07:59:48.649244 (+ 21279us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 07:59:48.649518 (+ 274us) raft_consensus.cc:1532] Early marking committed up to index 7
0831 07:59:48.743984 (+ 94466us) log.cc:844] Serialized 14 byte log entry
0831 07:59:48.869296 (+125312us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 07:59:48.869353 (+ 57us) raft_consensus.cc:1643] Marking committed up to 7
0831 07:59:48.869400 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 07:59:48.869460 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 07:59:48.869577 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":378,"prepare.queue_time_us":112678,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":9091,"thread_start_us":182565,"threads_started":2}
I20250831 07:59:48.865142 32469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d510d241da34d25a50f2d65d17041a0" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7ecb257174554a26b2c35ad04b0eabae" is_pre_election: true
I20250831 07:59:48.878515 32436 leader_election.cc:304] T a187e6c5ed8f4214b4486aab25955909 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ecb257174554a26b2c35ad04b0eabae; no voters: 8247a38b550649aa9cd6e7668dcaec10, a0d845d01c8c4894895f66e3beae298c
I20250831 07:59:48.878890 31879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51246 (request call id 355) took 2205 ms. Trace:
I20250831 07:59:48.835743 2569 leader_election.cc:290] T f2f76351ade3435092b6fac201772424 P a0d845d01c8c4894895f66e3beae298c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:48.858049 2583 raft_consensus.cc:2747] T 6d77811606c643a4970bc6b8a5f39fdb P 7ecb257174554a26b2c35ad04b0eabae [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250831 07:59:48.845182 2286 leader_election.cc:290] T 6b2e336f43934f118ecacdc1e1d2da60 P 8247a38b550649aa9cd6e7668dcaec10 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 7ecb257174554a26b2c35ad04b0eabae (127.29.217.4:43319)
I20250831 07:59:48.880834 31800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "660e7b2f56414f4fbd04a1929c99b390" candidate_uuid: "a0d845d01c8c4894895f66e3beae298c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7512ba2ed3a94982a22a75f51a647e65" is_pre_election: true
I20250831 07:59:48.929075 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b32e93c114645eeb4e707778c91e12e" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:48.940799 31730 raft_consensus.cc:2466] T 2b32e93c114645eeb4e707778c91e12e P a0d845d01c8c4894895f66e3beae298c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ecb257174554a26b2c35ad04b0eabae in term 9.
I20250831 07:59:48.941497 2514 leader_election.cc:290] T fcbe3794f3fc40819ffbbfef02a071c3 P 7ecb257174554a26b2c35ad04b0eabae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7512ba2ed3a94982a22a75f51a647e65 (127.29.217.2:37261), 8247a38b550649aa9cd6e7668dcaec10 (127.29.217.3:42585)
I20250831 07:59:48.941845 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e12d332258d4d0b8d0421b49ed5d0c0" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:48.942152 2514 consensus_queue.cc:1046] T 719b7e1bd5624bb9a08892e34c5e1b47 P 7ecb257174554a26b2c35ad04b0eabae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7512ba2ed3a94982a22a75f51a647e65" member_type: VOTER last_known_addr { host: "127.29.217.2" port: 37261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250831 07:59:48.943154 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2667d2ec934433ac10d395c1d29db4" candidate_uuid: "8247a38b550649aa9cd6e7668dcaec10" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d845d01c8c4894895f66e3beae298c" is_pre_election: true
I20250831 07:59:48.943765 31730 raft_consensus.cc:2466] T bb2667d2ec934433ac10d395c1d29db4 P a0d845d01c8c4894895f66e3beae298c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8247a38b550649aa9cd6e7668dcaec10 in term 5.
I20250831 07:59:48.944537 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90effb264f51485dac220ee32f49e4ad" candidate_uuid: "7ecb257174554a26b2c35ad04b0eabae" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "